[{"data":1,"prerenderedAt":341},["ShallowReactive",2],{"breadcrumb-blog-post":3,"solutions-page-es":175},{"id":4,"title":5,"author":6,"body":7,"category":160,"date":161,"description":162,"extension":163,"featured":164,"geo":6,"image":165,"manual_override":164,"meta":166,"navigation":167,"path":168,"readTime":169,"schema":6,"section_hashes":6,"seo":170,"sitemap":171,"source_hash":6,"source_locale":6,"stem":172,"tier":173,"tier_1_approved":164,"tier_1_approved_at":6,"tier_1_approved_by":6,"tier_1_deadline":6,"tier_1_reviewer":6,"translated_at":6,"translated_from_hash":6,"translation_model":6,"translation_provider":6,"translation_status":6,"__hash__":174},"blog/blog/2022-02-08-data-pressure.md","Dealing with data-pressure in message-based systems",null,{"type":8,"value":9,"toc":152},"minimark",[10,15,19,22,34,37,41,44,47,50,60,63,66,88,95,106,112,116,119,125,129],[11,12,14],"h2",{"id":13},"what-is-data-pressure","What is data-pressure?",[16,17,18],"p",{},"You hear a lot about data-pressure when it comes to non-stop systems. What is it and why is it so important?",[16,20,21],{},"Pressure in the physical sense describes an imbalance between gas or fluid between two confined compartments. It goes both ways until an equilibrium is reached. If you want to manage it, you usually put a valve between the two.",[16,23,24,25,29,30,33],{},"In data processing systems ",[26,27,28],"strong",{},"data-pressure",", or ",[26,31,32],{},"upstream-pressure"," describes the amount of data which is ready for processing. For file based (batch) solutions this simply describes the amount of files which are waiting to be processed (asynchronous). Processing speed is purely based on the processing power of the downstream actors and is demand based. Data-pressure in batch usually is no threat to system overload because it is implicit. The batch processing system will always only process as much as it can.",[16,35,36],{},"It's a very different story in modern, message-driven, real-time processing environments, however, where data-pressure is explicit because data needs to be processed as it arrives.",[11,38,40],{"id":39},"importance-of-back-pressure-in-non-stop-real-time-systems","Importance of back-pressure in non-stop real-time systems",[16,42,43],{},"Message-driven use-cases usually require that data should be handled in real-time, at all times. Therefore, systems have to be able to scale elastically in order to handle peak loads, or free up unneeded resources during low data-pressure windows.",[16,45,46],{},"There are countless examples of architectures which clog-up when dealing with large data loads. This often results in a vicious-cycle which typically leads to cardiac-arrest of such an architecture. The main conundrum is a missing negative demand-signal (or high data back-pressure-signal) to upstream actors upon which they can react. If there were such a signal, appropriate actions could be taken.",[16,48,49],{},"Such actions could be:",[51,52,53,57],"ul",{},[54,55,56],"li",{},"slowing processing down overall up the data stream, or",[54,58,59],{},"spawning more processing power to handle the added pressure",[16,61,62],{},"Once upstream data-pressure decreases, the counter-measures can be reversed. More data can again be delivered, or previously activated processing power can be decommissioned.",[16,64,65],{},"So to summarize, we have:",[67,68,69,79],"ol",{},[54,70,71,72,75,76,78],{},"a ",[26,73,74],{},"data-signal"," or ",[26,77,32],{}," which signals that data is available for processing, and we have",[54,80,71,81,75,84,87],{},[26,82,83],{},"demand-signal",[26,85,86],{},"back-pressure"," which signals how loaded the downstream actors are, and whether pressure from upstream actors can be relieved on to downstream actors.",[16,89,90],{},[91,92],"img",{"alt":93,"src":94},"Data-pressure Reactivity","/images/blog/2022-02-08/e436ddd5.png",[16,96,97,98,105],{},"Using these signals, the system is able to negotiate an equilibrium between all participants which ensures that processing never stops, but rather slows down (or additional capacity is automatically made available). This problem is well recognized and defined in the ",[99,100,104],"a",{"href":101,"rel":102},"https://www.reactivemanifesto.org/",[103],"nofollow","Reactive Manifesto"," which requires systems to be message-driven, elastic and resilient, and therefore responsive to load. Systems which cater to these requirements are called \"Reactive\".",[16,107,108],{},[91,109],{"alt":110,"src":111},"Reactive Manifesto: Means - Form - Value","/images/blog/2022-02-08/f8cdb1ba.png",[11,113,115],{"id":114},"how-laylineio-handles-it","How layline.io handles it",[16,117,118],{},"It sounds like the solution to the back-pressure challenge is simple. But it's actually hard to solve since all participants in this dance need to be data-pressure aware, both ways. Reactive stream management has solved this problem which is why layline.io takes full advantage it under the hood. It's not for the faint-of-heart, however, and comes with a steep learning and experience curve attached. layline.io shields its users from this complexity in an easy-to-use platform, which provides all production necessary features like UI-driven low-code configurability, one-click deployment, monitoring and much more.",[16,120,121],{},[91,122],{"alt":123,"src":124},"layline.io Project Configuration","/images/blog/2022-02-08/project_workflow_04.webp",[11,126,128],{"id":127},"resources","Resources",[51,130,131,136,145],{},[54,132,133],{},[99,134,104],{"href":101,"rel":135},[103],[54,137,138,139,144],{},"Read more about layline.io ",[99,140,143],{"href":141,"rel":142},"https://layline.io/",[103],"here",".",[54,146,147,148,144],{},"Contact us at ",[99,149,151],{"href":150},"mailto:hello@layline.io","hello@layline.io",{"title":153,"searchDepth":154,"depth":154,"links":155},"",2,[156,157,158,159],{"id":13,"depth":154,"text":14},{"id":39,"depth":154,"text":40},{"id":114,"depth":154,"text":115},{"id":127,"depth":154,"text":128},"Article","2022-02-08","How to deal with data pressure in non-stop message-driven solutions and ensure non-stop uptime under load.","md",false,"/images/blog/2022-02-08/lucas-van-oort-_FjIWDrtfmU-unsplash.webp",{},true,"/blog/2022-02-08-data-pressure","4 min",{"title":5,"description":162},{"loc":168},"blog/2022-02-08-data-pressure","2","x0f2ib8nEa9ZPX713ZxgzY6LLnFGIWWB6pktWteME5k",{"doc":176,"isFallback":164,"effectiveLocale":340},{"source_locale":177,"source_hash":178,"translation_status":179,"translated_at":180,"manual_override":167,"title":181,"description":182,"hero":183,"roles":201,"industries":240,"workflows":278,"quickLinks":314,"finalCta":334,"body":153},"en","ecb8a831fa12df7fdb6bf93c1d5d028c7f22326f0f472192ce09a21cc07818c0","up_to_date","2026-06-18T12:30:00.000Z","Soluciones | layline.io - Integración de datos por rol e industria","Descubre cómo layline.io resuelve los desafíos de integración de datos para distintos roles e industrias. Desde ingenieros de datos hasta equipos financieros, encuentra el enfoque adecuado.",{"badge":184,"titlePrefix":187,"titleHighlight":188,"description":189,"backgroundImage":190,"primaryCta":193,"secondaryCta":197},{"label":185,"icon":186},"Soluciones","i-ph-lightbulb","Soluciones para cada","rol e industria","Tanto si eres un ingeniero de datos que crea canalizaciones como si formas parte de un equipo financiero que procesa transacciones, layline.io se adapta a tus necesidades.",{"src":191,"alt":192},"/images/unsplash/photo-1504384308090-c894fdcc538d.jpg","Fondo de la página de soluciones",{"label":194,"to":195,"icon":196},"Empieza gratis","/get-started","i-ph-rocket",{"label":198,"to":199,"icon":200},"Reserva una demo","/resources/booking","i-ph-calendar",{"badge":202,"titlePrefix":205,"titleHighlight":206,"description":207,"linkLabel":208,"cards":209},{"label":203,"icon":204},"Por rol","i-ph-users","Diseñado para","tu equipo","Soluciones pensadas para profesionales de datos en todos los niveles.","Saber más",[210,218,226,233],{"title":211,"description":212,"icon":213,"href":214,"color":215,"image":216,"imagePosition":217},"CTOs y responsables de ingeniería","Toma decisiones tecnológicas estratégicas con una visión clara del coste total de propiedad, menos dependencia del proveedor y una mejor evaluación entre construir o comprar.","i-ph-briefcase","/solutions/ctos","orange","/images/photos/notebook-mockup-on-a-ladys-lap.jpeg","top center",{"title":219,"description":220,"icon":221,"href":222,"color":223,"image":224,"imagePosition":225},"Ingenieros de datos","Crea y opera canalizaciones de datos robustas con flujos de trabajo visuales y capacidades de procesamiento en tiempo real.","i-ph-database","/solutions/data-engineers","blue","/images/unsplash/photo-1551434678-e076c223a692.jpg","center center",{"title":227,"description":228,"icon":229,"href":230,"color":231,"image":232,"imagePosition":225},"Ingenieros de plataforma","Despliega infraestructura de datos escalable con flujos de trabajo nativos de Kubernetes y observabilidad de nivel empresarial.","i-ph-computer-tower","/solutions/platform-engineers","violet","/images/photos/austin-distel-mpN7xjKQ_Ns-unsplash_medium.jpeg",{"title":234,"description":235,"icon":236,"href":237,"color":238,"image":239,"imagePosition":225},"Ingenieros de analítica","Convierte datos en bruto en conjuntos listos para analítica con potentes capacidades de transformación.","i-ph-chart-bar","/solutions/analytics-engineers","teal","/images/unsplash/photo-1551288049-bebda4e38f71.jpg",{"badge":241,"titlePrefix":244,"titleHighlight":245,"description":246,"linkLabel":247,"cards":248},{"label":242,"icon":243},"Por industria","i-ph-building","Soluciones para la","industria","Integración de datos adaptada a los retos específicos de tu industria.","Explorar",[249,255,260,266,272],{"title":250,"description":251,"icon":252,"href":253,"color":254},"Finanzas","Procesamiento de transacciones en tiempo real, informes de cumplimiento y analítica de riesgo.","i-ph-bank","/solutions/finance","emerald",{"title":256,"description":257,"icon":258,"href":259,"color":215},"Comercio electrónico","Sincronización de inventario, procesamiento de pedidos y analítica de clientes a escala.","i-ph-shopping-cart","/solutions/ecommerce",{"title":261,"description":262,"icon":263,"href":264,"color":265},"Manufactura","Integración de datos IoT, control de calidad y visibilidad de la cadena de suministro.","i-ph-factory","/solutions/manufacturing","slate",{"title":267,"description":268,"icon":269,"href":270,"color":271},"Hostelería","Integración de datos de huéspedes, sistemas de reservas y analítica operativa.","i-ph-bed","/solutions/hospitality","rose",{"title":273,"description":274,"icon":275,"href":276,"color":277},"Telecomunicaciones","Procesamiento de datos de red, análisis de CDR y optimización de la experiencia del cliente.","i-ph-broadcast","/solutions/telecom","sky",{"badge":279,"titlePrefix":282,"titleHighlight":283,"description":284,"linkLabel":208,"cards":285},{"label":280,"icon":281},"Por flujo de trabajo","i-ph-flow-arrow","Patrones de","flujo de trabajo","Patrones de flujo de trabajo listos para usar en escenarios comunes de integración de datos.",[286,291,297,303,309],{"title":287,"description":288,"icon":281,"href":289,"color":290},"Streaming reactivo","Procesa millones de eventos por segundo con una gestión inteligente de la backpressure.","/solutions/reactive-streaming","indigo",{"title":292,"description":293,"icon":294,"href":295,"color":296},"Procesamiento en el edge","Procesa datos en el edge con más del 90 % de ahorro de ancho de banda.","i-ph-cpu","/solutions/edge-processing","amber",{"title":298,"description":299,"icon":300,"href":301,"color":302},"Detección de fraude","Detección de anomalías en tiempo real con reglas personalizadas en Python y JavaScript.","i-ph-shield-warning","/solutions/fraud-detection","red",{"title":304,"description":305,"icon":306,"href":307,"color":308},"Orquestación de datos","Orquestación y coordinación inteligentes entre múltiples fuentes de datos.","i-ph-git-merge","/solutions/data-orchestration","cyan",{"title":310,"description":311,"icon":221,"href":312,"color":313},"Pipelines ETL/ELT","Canalizaciones por lotes y en tiempo real en una plataforma unificada.","/solutions/etl-elt","lime",{"title":315,"description":316,"links":317},"¿No sabes por dónde empezar?","Te ayudamos a encontrar la solución adecuada.",[318,322,326,330],{"label":319,"to":320,"icon":321,"accent":223},"Visión general del producto","/product/overview","i-ph-play-circle",{"label":323,"to":324,"icon":325,"accent":231},"Comparar ediciones","/editions/comparison","i-ph-table",{"label":327,"href":328,"icon":329,"accent":308,"external":167},"Documentación","https://doc.layline.io","i-ph-book-open",{"label":331,"to":332,"icon":333,"accent":254},"Contactar con ventas","/resources/contact","i-ph-chat-circle",{"title":335,"description":336,"primaryCta":337,"secondaryCta":338},"¿Listo para transformar tus flujos de trabajo de datos?","Únete a los equipos que están construyendo canalizaciones de datos listas para producción con layline.io.",{"label":194,"to":195,"icon":196},{"label":339,"to":199,"icon":200},"Programar una demo","es",1787645790972]