ArcGIS Cloud-Native

ArcGIS Opportunities with Cloud-Native and Serverless Computing Technologies

GCS facilitates your journey to Cloud-Native with ArcGIS and Amazon Web Services (AWS) and Azure

Organizations are driven to Cloud-Native architectures by business needs.

  • Security – Strong identity, traceability, protecting data in transit and rest, and preparing for threats.
  • Reliability – Handling changes in demand with automation, automatically recovering from failure, and scaling horizontally to increase aggregate system availability.
  • Performance Efficiency – Serverless architectures, democratize advanced technologies, and saving cost by only using the right amount of resources while achieving performance, reliability.
  • Cost Optimization – Measuring overall efficiency, using managed services to reduce cost of ownership, and adopting a consumption model.
  • Operational Excellence – Performing operations as code, implementing DevOps to achieve desired business outcomes such as security, reliability, performance efficiency, and cost optimization.

GCS facilitates the assessment process for refactoring or re-architecting your existing system to utilize cloud-native features. Whether you moved to the cloud with a “lift-and-shift” of your existing on-premise architecture, performed some re-platforming, or not yet in the cloud, GCS can advise a path forward using Cloud-Native technologies to meet your business needs.

GCS helps you move away from monolithic architectures to Cloud-Native server-less architectures.

ArcGIS Cloud-Native Opportunities

The following are just a few examples of opportunities to use Cloud-Native technologies with ArcGIS.

ArcGIS Server Elastic Scaling

In this architecture, two or more ArcGIS Server sites are configured behind Amazon Elastic Load Balancer (ELB) to increase the capacity of ArcGIS Server deployment.

Adding Auto Scaling to your ArcGIS application architecture is one way to maximize the benefits of the Cloud. This is an easy way to scale ArcGIS horizontally improving reliability, availability, and reducing total cost of ownership.

When you use Amazon EC2 Auto Scaling, your applications gain the following benefits:

Better fault tolerance

Amazon EC2 Auto Scaling can detect when an instance is unhealthy, terminate it, and launch an instance to replace it.

You can also configure Amazon EC2 Auto Scaling to use multiple Availability Zones. If one Availability Zone becomes unavailable, Amazon EC2 Auto Scaling can launch instances in another one to compensate.

Fault tolerance across AWS regions. If you require even stronger fault tolerance beyond Availability Zones, Amazon Route 53 allows low latency routing across AWS regions. With Amazon Route 53 Latency Based Routing (LBR) feature, instances in several AWS regions can be configured and have requests from end-users automatically routed to the region with the lowest latency.

Better availability

Amazon EC2 Auto Scaling helps ensure that your application always have the right amount of capacity to handle the current traffic demand.

For instance with ArcGIS Server, Auto Scaling allows you to spin up new instances of ArcGIS Server based on average RAM or CPU utilization thresholds across your instance stack. It is known that a number of ArcSOC.exe instances can consume high memory (RAM) resources, causing ArcGIS performance issues. Instead of making trade-offs with ArcGIS Server configuration such as setting services to use the shared instance pool or setting the minimum number of instances to ‘0’, use EC2 Auto Scaling to meet performance and capacity needs. The ArcGIS Server configuration fixes come at operational costs, i.e. GIS services sharing individual ArcSOC.exe processes may fail based on other GIS service failures. And, setting the minimum number of instances to zero may result in delayed performance as it will take time for new ArcSOCs to spin up after a request. There is a reason why these fixes are not the default ArcGIS Server configuration settings.

Better cost management

Amazon EC2 Auto Scaling can dynamically increase and decrease capacity as needed. Because you pay for the EC2 instances you use, you save money by launching instances when they are needed and terminating them when they aren’t.

Elastic Load Balancing distributes incoming application or network traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses, in multiple Availability Zones. Elastic Load Balancing scales your load balancer as traffic to your application changes over time. It can automatically scale to the vast majority of workloads.

ArcGIS Server High Availability

For high availability ArcGIS Enterprise architectures, storing ArcGIS Server configuration store in AWS DynamoDB or S3 takes advantage of cloud-native offerings.

Amazon S3 can be employed to store any type of object which allows for uses like storage for Internet applications, data backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It’s a fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching for internet-scale applications.

Multiple EC2 instances with ArcGIS Server may access the same files stored in an Amazon S3 bucket or DynamoDB.

Optimize Map Cache Storage and Delivery

Use Amazon S3 for large map caches and Amazon CloudFront to deliver quickly.

You can configure ArcGIS Server to build and store map cache tiles on Amazon S3. Then, using Amazon CloudFront, push your tile services to CDNs for super fast delivery. If your map caches do not change frequently, this is an excellent option. With Amazon S3, you only pay for what you use. It is an ideal service for backing up caches, as well as any type of data.

CloudFront delivers your content through a worldwide network of data centers called edge locations. When a user requests content that you’re serving with CloudFront, the user is routed to the edge location that provides the lowest latency (time delay), so that content is delivered with the best possible performance.

Amazon S3 can be employed to store any type of object which allows for uses like storage for Internet applications, data backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

Cloud Native Geodatabase

Moving your Enterprise Geodatabase to Amazon RDS has numerous advantages. Amazon RDS is easy to administer, highly scalable, available, durable, fast, secure, and inexpensive.

Amazon RDS supported Geodatabase RDMS

Supported RDS databases that also support ArcGIS geodatabases include: Microsoft SQL Server, PostgreSQL, and Amazon Aurora (PostgreSQL-Compatible Edition).

Unfortunately, it looks like Oracle is not supported currently by Esri.

Amazon RDS Geodatabase Anti-Patterns

With RDS, you no longer have full control or access over the database. If you need full access, then you need to manage your database on an EC2 instance or keep it on-premise.

If you are using a non-supported Geodatabase RDS, then you need to manage your database on an EC2 instance or keep it on-premise. Again, supported RDS databases that also support ArcGIS geodatabases include: Microsoft SQL Server, PostgreSQL, and Amazon Aurora (PostgreSQL-Compatible Edition).

Geodatabases in AWS instances are not intended to be accessed directly from on-premises ArcGIS clients as performance will be far slower than when the geodatabases are accessed from ArcGIS clients on AWS.

ArcGIS Enterprise in the cloud, Geodatabases on Amazon Web Services, Esri Online documentation
A GIS Server site on AWS with ArcGIS Server and an Amazon RDS instance, with two additional GIS Server installations on AWS instances available when CPU usage exceeds a specified threshold.

Given the statement above, you can have an efficient hybrid cloud where on-premise ArcGIS and Amazon RDS Geodatabase instances deliver reliable high performance user experiences. The ArcGIS online documentation is assuming an Internet connection. Using AWS Direct Connect, you can establish private connectivity between AWS and an on-premise datacenter, office, or colocation environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections. With AWS Direct Connect, you can create a high performance user experience with a hybrid cloud architecture.

If you have GIS Analysts using ArcGIS desktop apps as part of your architecture, there are several AWS technologies to bring them into the cloud eliminating any network performance issues.

Amazon WorkSpaces is a managed, secure Desktop-as-a-Service (DaaS) solution. You can use Amazon WorkSpaces to provision either Windows or Linux desktops in just a few minutes and quickly scale to provide thousands of desktops to workers across the globe. You can pay either monthly or hourly, just for the WorkSpaces you launch, which helps you save money when compared to traditional desktops and on-premises VDI solutions.

Lastly, at this time, ArcGIS supports only database-authenticated connections to AWS RDS Geodatabase instances.

Reasons to use Amazon RDS with an ArcGIS Geodatabase

RDS is a managed service where you do not have access to underlying EC2 instances (no root access) database access. But, you get the following fully managed service:

  • Security and patching of the DB instances.
  • Automated backup.
  • Software updates.
  • Easy scaling for storage and compute.
  • Multi-AZ option with synchronous replication.
  • Automatics failover for Multi-AZ option.
  • Read replicas option for read heavy workloads.

High Availability Approaches for AWS hosted ArcGIS Geodatabases

For High Availability (HA) architectures, Amazon Aurora is the best option over other Esri supported databases. Aurora is an AWS proprietary database which has far more cloud-native features baked into the AWS ecosystem. Two copies of data are kept in each Availability Zone (AZ) with a minimum of three AZ’s; totaling six copies. Aurora can handle the loss of up to two copies of data without affecting DB write availability and up to three copies without affecting read availability. Aurora has cross-regional read replicas to improve disaster recovery posture, scale read operations in regions closer to your application users, and easily migrate from one region to another. Aurora can be enabled to span multiple AWS regions with the Global Database feature. The Multi-Master feature adds the ability to scale out read/write workloads across multiple AZs. Aurora is high performance with low cost.

If you use a different database, then you must enable Multi-AZ RDS for High Availability.

Multi-AZ RDS creates a replica in another AZ and synchronously replaces to it for disaster recovery only.

If the database runs on EC2, you must design the High Availability yourself.

Because they provide enhanced availability, Amazon recommends you use multi-AZ deployments for production systems. Be aware, though, that these deployments cost more to use per hour and for storage than other deployments. Consult the Amazon RDS pricing guide for more information.

ArcGIS Enterprise in the cloud, Geodatabases on Amazon Web Services, Esri Online documentation

Multi-AZ and Read Replicas are used for high availability, fault tolerance and performance scaling. Like the Esri online documentation states, multi-AZ deployments are more expensive. That said, the Total Cost of Ownership (TCO) is much lower than implementing the same architecture on-premise; not too mention, less painful.

In addition, with multi-AZ, Reserved (vs On-Demand) instance payment plans are available to reduce costs. Reserved instances are based on DB engine, DB instance class, license model, and region. They can be moved between Availability Zones (AZ) in the same region. They can be applied to Read Replicas if DB instance class and region are the same.

Lastly, keep in mind DynamoDB. With regard to HA, it has better HA over RDS because of inherent fault tolerance. Although not compatible with ArcGIS Geodatabase, consider DynamoDB as a non-spatial data store. DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.

Amazon RDS Read Replicas and ArcGIS

Amazon RDS Read Replicas provide enhanced performance and durability for RDS database (DB) instances. They make it easy to elastically scale out beyond the capacity constraints of a single DB instance for read-heavy database workloads. You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Read replicas can also be promoted when needed to become standalone DB instances.

Read replicas are used for read-heavy DBs and replication is asynchronous. Read Replicas are for workload sharing and offloading.

ArcGIS Geodatabase compatible Read Replicas are available in Amazon RDS for PostgreSQL, SQL Server, and Amazon Aurora.

Amazon Aurora Serverless Geodatabase

Depending on your Use Case, Aurora Serverless can be more cost-effective than RDS while still meeting your compute and memory capacity needs.

Amazon Aurora Serverless is an on-demand, autoscaling configuration for Amazon Aurora. It is an alternative to RDS and may give greater benefits with regard to cost savings and simplicity. Aurora Serverless removes much of the complexity of managing DB instances and capacity. Yet, Aurora Serverless seamlessly scales and uses the same fault-tolerant, distributed storage system with six-way replication as Aurora (RDS) to protect against data loss.

You may consider Aurora Serverless for your geodatabase implementation in several cases such as infrequently used or new apps, variable or unpredictable workloads, dev and test environments, or multi-tenant apps. With Aurora Serverless, you pay for only the database resources that you consume on a per-second basis.

Geodatabase Migration to AWS

AWS Database Migration Service helps you migrate database to AWS quickly and securely. Use along with the Schema Conversion Tool (SCT) to migrate databases to AWS RDS or EC2-based databases.

Esri has the following help on strategies for loading data into a geodatabase and data transfer.

AWS RDS Related Services

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.

AWS Database Migration Service helps you migrate databases to AWS quickly and securely.

Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud, that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases.

Fast Content Delivery for Static Mapping Web Apps

You can host your static web mapping apps in S3 and use CloudFront for the fastest and most scalable architecture. Hosting your web apps in Amazon S3 relieves the burden of managing and paying for a web server. The cost is negligible.

Amazon S3 can be employed to store any type of object which allows for uses like storage for Internet applications, data backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

Serverless Compute Service for ArcGIS Workloads

Migrating your GIS and business workloads to AWS Lambda eliminates servers (and server environments) to automatically scale, increase speed, and provide consistent performance. And, the cost is very low. You pay only for the compute time you consume.

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.

Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale. You can create a web API with an HTTP endpoint for your Lambda function by using Amazon API Gateway. 

ArcGIS Distributed Analytics with Structured and Unstructured Storage

Challenges associated with Deep Learning and Big Data include: managing enormous amounts of structured and unstructured data, and getting the data as close as possible to the analytics for optimal performance. Cloud-Native services that facilitate Data Lakes enable this process.

Learn about Microsoft Azure Data Lakes

A data lake is a centralized, curated, and secured repository that stores all your data, both in its original form and prepared for analysis. They can be attached to multiple cloud server instances.

You can expose AWS Lake Formation data mainly via Amazon Athena, Amazon Redshift, and Amazon EMR.

Both Athena and Redshift support geometry data types; no geodatabase support. It appears Athena supports a Java ESRI Geometry API.

By far, EMR looks like the best (and easiest) solution using Apache Hadoop. Folks have been creating big data analysis solutions with Esri and Hadoop for a couple years.

GIS Tools for Hadoop works with big spatial data (big data with location) and allows you to complete spatial analysis using the power of distributed processing in Hadoop.

The GIS Tools for Hadoop toolkit allows you to leverage the Hadoop framework to complete spatial analysis on spatial data; for example:

  1. Run a filter and aggregate operations on billions of spatial data records based on location.
  2. Define new areas represented as polygons, and run a point in polygon analysis on billions of spatial data records inside Hadoop.
  3. Visualize analysis results on a map and apply informative symbology.
  4. Integrate your maps in reports, or publish them as map applications online.

In some scenarios, you can expose S3 data via Storage Gateway to mount drives on EC2 instances running ArcGIS.

AWS Lake Formation is a service that makes it easy to set up a secure data lake in a matter of days.

Amazon S3 can be employed to store any type of object. This allows for uses like storage for Internet applications, data backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.

Scalable, Fully Managed Elastic File System for Image Server

AWS EFS and ArcGIS Image Server
Easily manage petabytes of imagery data for ArcGIS Image Server (Linux) across AWS Regions and Availability Zones. Pay only for what you use (elastic); unlike EBS storage where you pay for provisioned space. EFS has 10+ GB per second throughput; while EBS provisioned IOPS has up to 2 GB per second.

One challenge for implementing a robust ArcGIS Image Server architecture is managing imagery file-based data. ArcGIS Image Server allows you to assemble, process, analyze, and manage large collections of overlapping, multiresolution imagery and raster data from different sensors, sources, and time periods. It does this well. But, collecting, updating, adding, or managing raster file data on storage devices can be challenging and costly with large amounts of data. This is especially true for scenarios whereby you are continuously collecting (growing) your raster file footprint, or temporarily creating and disposing of large raster datasets through big data analysis, etc.

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed, elastic NFS file system for use with AWS Cloud services and on-premises resources. It is built to scale on demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

Using Amazon EFS with ArcGIS Image Server on Linux saves maintenance time and costs by growing and shrinking automatically as you add and remove files. This results in profound cost savings as your GIS analysts no longer need to work with Cloud DevOps teams to continually analyze disk space, provision, decommission, and manage capacity. It is more efficient since you are paying for only the capacity that you are using, as opposed to allocating extra space in anticipation of future growth, while hoping that disk capacity calculations were correct and requirements do not change.

AWS DataSync is a data transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon S3 or Amazon Elastic File System. The service enables one-time data migrations, recurring data processing workflows, and automated replication for data protection and recovery.

Continuously Delivered Scalable GeoServices on Kubernetes

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.

Internally, Esri uses Kuberbetes (not necessarily in AWS) for DevOps, allowing them to build highly scalable services and iterate on them rapidly. Kubernetes provides all the abstractions necessary to manage a database, scale stateless apps, run cron jobs and handle network ingress. Kubernetes helps small teams scale to many users and different microservices without sacrificing velocity.

Esri is deploying ArcGIS Analytics for IoT with Kubernetes to provide high performance, reliability, and security at massive scales.

ArcGIS Notebook Server uses Docker container allocation software to provide security and isolate resources for notebook authors. A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

ECS is an AWS-specific platform that supports Docker containers whereby EKS is compatible with Kubernetes. EKS is a good choice for “lift and shift” from other Kubernertes deployements. ECS is considered simpler to learn and use while EKS is considered more feature-rich and complex with a steep learning curve. ECS leverages AWS services like Route 53, ALB, and CloudWatch while EKS hosts many extensions from the Kubernetes platform via a wide variety of third-party and community add-ons.

With ECS, you can choose to run your ECS clusters using AWS Fargate, which is serverless compute for containers. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design.

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.

Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes service.

AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). 

Large GIS, Raster, and System Data Transfers for Cloud Migration or On-Going Operations

Moving large GIS datasets to the cloud can be a challenge. Typically, these are raster type datasets.  Establishing an AWS Direct Connect allows data to be transferred directly and quickly through dedicated physical connection.  If the data needing to be transferred is for a one-time migration or not very frequently, AWS import/export services can be used to ship hard-drives.  For very large datasets, AWS Snowball is a rugged physical device which can be used for data transfer and much more.   

AWS Direct Connect makes it easy to establish a dedicated connection from an on-premises network to Amazon VPC. Using AWS Direct Connect, you can establish private connectivity between AWS and your data center, office, or collocated environment. This private connection can reduce network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based connections.

AWS Direct Connect makes it easy to establish a dedicated connection from an on-premises network.

VM Import/Export enables you to easily import virtual machine images from your existing environment to Amazon EC2 instances and export them back to your on-premises environment. This offering allows you to leverage your existing investments in the virtual machines that you have built to meet your IT security, configuration management, and compliance requirements by bringing those virtual machines into Amazon EC2 as ready-to-use instances. You can also export imported instances back to your on-premises virtualization infrastructure, allowing you to deploy workloads across your IT infrastructure.

AWS Snowball uses secure, rugged devices so you can bring AWS computing and storage capabilities to your edge environments, and transfer data into and out of AWS. The service delivers you Snowball Edge devices with storage and optional Amazon EC2 and AWS IOT Greengrass compute in shippable, hardened, secure cases. With AWS Snowball, you bring cloud capabilities for machine learning, data analytics, processing, and storage to your edge, for migrations, short-term data collection, or even long-term deployments. AWS Snowball devices work with or without the internet, do not require a dedicated IT operator, and are designed to be used in remote environments.

Migrating Data Across Regions and Accounts. A VPC Peering connection is a networking connection between two VPCs that enables you to route traffic between them. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different regions. This is a flexible option for sharing large datasets across AWS accounts.

ArcGIS Pro App Streaming

Amazon AppStream 2.0 is a fully managed application streaming service. You centrally manage your desktop applications on AppStream 2.0 and securely deliver them to any computer. You can easily scale to any number of users across the globe without acquiring, provisioning, and operating hardware or infrastructure.

Esri published a blog showing how to use Amazon AppStream 2.0 to deliver ArcGIS Pro. To get started using ArcGIS Pro with Amazon AppStream 2.0, please review the Amazon AppStream 2.0: Esri ArcGIS Pro Deployment Guide.

ArcGIS Cloud Certified Professionals

GCS’s team is comprised of certified ArcGIS and cloud professionals who are solution architects, system integrators, native cloud developers, data scientists, and professional project managers.

Esri Release Ready
As an Esri partner in the Release Ready Specialty, GCS is recognized for keeping pace with Esri technology, expertise, and the community by helping users make smart decisions using Web GIS.

Contact GCS

To learn more, contact us to speak to a certified ArcGIS Cloud Expert.

Your Data. Smarter.
GCS is a Geospatial Information Technology Services Company

search previous next tag category expand menu location phone mail time cart zoom edit close