AWS US-WEST-2 (Oregon) and US-EAST-1 issues

AWS reports Incident resolved. Thanks for your patience.

update from AWS: Sep 18 2:22 PM PDT We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. While we continue to make progress in addressing the issue, we wanted to provide some more details on the issue. Within Amazon Virtual Private Cloud (VPC) any changes to the network configuration - including launching an EC2 instance, attaching an Elastic IP address or Elastic Network Interface - needs to be propagated to the underlying hardware to ensure that network packets can flow between source and destination. We call this network configuration “network mappings”, as it contains information about network paths or mappings. Starting at 10:00 AM PDT this morning, we have been experiencing a delay in the propagation of these mappings within a single cell (part of the Availability Zone) in usw2-az1 and usw2-az2 Availability Zones. The root cause appears to be increased load to the subsystem responsible for the handling of these network mappings. We have been working to reduce the load on this service to improve propagation times, but while we have made some progress, mapping propagation latencies have not returned to normal levels. We continue to work to identify all forms of resource contention that could be leading to load, and have a few additional updates that we are currently working on. Once we have reduced the load on the subsystem responsible for network mapping propagation, we would expect full recovery. Since the issue only affects networking mappings as a result of changes to the network configuration, avoiding any mutating changes to your network configuration in the affected Availability Zone would limit impact. For customers looking to fail away from the affected Availability Zone or Region, please ensure that you are not relying on mutating network configuration in the affected Availability Zones as part of that process. We will continue to keep you updated as we make progress towards full recovery.

AWS has reported that a network propagation issue is impacting assets in Oregon and N. Virginia. These issues occur upon attempting to change network configs or launch new assets; currently running assets seem to not be impacted. More below from AWS: Sep 18 1:11 PM PDT We continue to make progress towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. The issue affects some instances in each of these Availability Zones, where network mappings are not being propagated to the underlying hardware. Any mutations (or changes) to the networking configurations - including launching an EC2 instance, attaching and Elastic IP address or Elastic Network Interface - would see delayed propagation times, which will affect network connectivity until completed. Other networking services like PrivateLink and API Gateway, are also seeing network connectivity issues where network mappings have been mutated. For other AWS services, such as Lambda and Container Services (ECS/EKS), delays in function and container creation, as well as increased error rates, may occur due to this issue. We continue to see improvements in the usw2-az1 Availability Zone and continue to work on resolving the issue in the usw2-az2 Availability Zone. We recommend that customers avoid making mutating changes within the affected Availability Zones, which will ensure that no new network mappings need to be propagated to the underlying hardware, avoiding impact from this issue. We will continue to keep you updated as we make progress towards full recovery.