Climatology Research for Canary Observatories at IAC: Automating Cold-Storage Data Access and Processing for Large-Scale Simulations on AWS
Key Challenges
At the Instituto de Astrofísica de Canarias (IAC), the Sky-quality team working on Climatology data analysis needed to run large-scale climatology simulations against historical datasets held in cold AWS storage (S3 Glacier). They had to define a process to bring that archived data back into an accessible tier, size the compute resources to run the data processing workloads, and manage the lifecycle and cost of the resulting data once processed.
Key Results
With Ankercloud, IAC automated climatology data retrieval, researcher notifications, parallel processing on AWS EC2, and storage cost optimization. Combined with right-sized computing and CloudWatch monitoring, the solution enables reliable, multi-hour simulations without manual data handling.
Overview
The Instituto de Astrofísica de Canarias (IAC), recognized by the Spanish government as a “Severo Ochoa Centre of Excellence,” is a nationally funded research centre that operates two of the world's leading astronomical observatories. Its mission is to advance research into astrophysics and related fields, develop and transfer associated technology, train scientific and technical staff, and promote collaboration with the national and international scientific community.
Within IAC, the team working on Climatology data analysis holds a large volume of historical data in Amazon S3 Glacier Deep Archive. To process the data, they were required to run simulations that could take several days to complete. Ankercloud was engaged to design and validate the underlying storage and infrastructure setup on AWS: data restoration from cold storage, lifecycle management, infrastructure provisioning, and operational monitoring. The final goal was to enable IAC's research team to run its climatology simulations reliably and cost-effectively, in a shorter time frame.
The primary goal of the climatology research was to perform a site-selection support analysis for any outcoming future facility to be installed in Canary Observatories. This research included 60 years (1960–2019) of key meteorological parameters—relative humidity, temperature, wind velocity, and wind direction—between the Teide Observatory (OT) and Roque de los Muchachos Observatory (ORM) using regional climate reanalysis data (UERRA).
Challenges
As the Sky-quality team at IAC looked to move from ad-hoc data handling toward a repeatable simulation workflow, several specific challenges needed to be addressed:
- Cold-storage accessibility: the datasets required for simulation runs sit in S3 Deep Archive, and no automated process existed to restore this data into S3 Standard storage or to notify researchers once it was ready to use.
- Compute sizing for long, parallel workloads: each simulation required running from several hours to days, opening the need to use multiple instances in parallel to speed up results.
- Lifecycle and cost management: once data was restored and processed, it needed a defined lifecycle policy so that storage costs did not accumulate unnecessarily over time.
- Operational visibility: there was no centralized health dashboard or alerting in place to monitor the infrastructure supporting the simulations.
Solution
The IAC team worked directly with Ankercloud’s tech team to implement and validate the process. The project was delivered as a Proof-of-Concept and included specific requirements discovery, storage and compute configuration, infrastructure-as-code script definition, and monitoring setup.
Key components of the solution include:
- Automated cold-storage restoration: a defined process restores data from S3 Deep Archive into S3 Standard storage and notifies the user once the data is available to be accessed, removing the need for manual retrieval requests.
- Lifecycle management: lifecycle policies govern the processed data, ensuring it is moved back to lower-cost storage tiers once it is no longer in active use.
- Right-sized compute architecture: the simulation environment is built on Amazon EC2 (c5.xlarge, Linux) with shared tenancy, sized to run up to 15 instances in parallel with 100 GB of EBS storage each, matching the Climatology team's computing requirements.
- Infrastructure-as-Code validation: the existing infrastructure provisioning script was reviewed, validated, and tested, alongside consultation on EC2 and storage configuration, to ensure a repeatable and reliable setup.
- Regional deployment: the architecture is deployed in AWS's Europe (Spain) region, with S3 Standard storage and CloudWatch monitoring configured alongside the EC2 compute layer.
The project was fully documented to ensure the team can operate the environment independently going forward.
Business Outcome
By validating the reliability of UERRA climate reanalysis against local weather station and radiosonde data, the research proved that regional reanalysis models can accurately model boundary layer and free atmosphere conditions for mountain observatory sites. This provides the broader scientific community with a robust, data-driven framework to inform final telescope site selection, design specifications, and operational planning without relying solely on ad-hoc local ground measurements.
The Proof-of-Concept gave IAC's Sky-quality team a validated foundation for running its data-intensive simulations on AWS, with clear benefits in the following domains:
- Scalability for Research Workloads: The compute architecture is available 24/7 and supports the team's typical pattern of 10–15 parallel EC2 instances, right-sized to match the optimal configuration required.
- Operational Efficiency: Automated data restoration and Infrastructure-as-Code provisioning removed manual data-retrieval steps and notify researchers as soon as data is ready to use.
- Cost Management: Lifecycle policies keep processed data in the appropriate storage tier over time, avoiding unnecessary long-term storage costs.
- Enablement and Knowledge Transfer: Full documentation and dedicated walkthrough sessions were delivered at the close of the engagement, enabling IAC's team to operate and extend the environment independently.
