The closest circuit breaker that we can relate to in our daily lives is MCB, which is present in every home to protect devices from electrical circuit failures which could lead to increased flow of current beyond a specific capacity. It achieves protection from high in-flow of current by disrupting or opening the circuit to stop the flow.
Let’s consider we have microservices A, B and C. Assume that A calls B and B further calls C. Now, if at any point of time, if C experiences some issues (like CPU usage is high, or service is down) due to which it takes a long time for responding. This can lead to issues in B as C’s response time has increased, which will lead to resources (threads) at B being occupied for a longer period of time. This will further slow down the responses from B to A also. Such failures are called cascading failures. So these cascading failures should be handled to prevent them from translating to other services and causing failures.
A cascading failure is a process in which the failure of a downstream service can trigger the failure of upstream services.
Circuit breaker provides a way to cut off the connection to other services if a failure threshold limit has been reached for that service and the connection is turned on after a specified cooling period which would give time to that service to recover
Circuit Breaker has 3 states:
CLOSE — connection to downstream service is allowed
OPEN — connection to downstream service is forbidden and we may handle such cases using a fallback method or default response
HALF OPEN — some calls to downstream services are allowed
State Cycle
Initially, the circuit breaker will be in a CLOSE state, i.e., all communication to downstream service is allowed. Once the failure threshold limit is reached for downstream service, the state changes to OPEN, i.e., communication to downstream service is forbidden/denied. After the cooling period, the state is changed to HALF OPEN, if any request comes for the downstream service, then only a certain number of calls are allowed. If the threshold success rate is achieved, the state changes to CLOSE, else it again moves to OPEN state, and again it needs to wait for the cooling period.
The OPEN state helps in preventing cascading failures by forbidding any requests to downstream service, thus not blocking any resources in the current service. Further, a fallback method can be defined to gracefully handle negative scenarios.
Popular libraries available for circuit breakers are :
Hystrix
Resilience4j
In another article, I have discussed how to build a basic circuit breaker. Please go through it.