What are your insights and guidelines for "Batch Modernization"? What are the target future architecture for Batch Workloads? Such as Data Streaming, Events, Data Mesh, AWS Batch? What are the criteria to chose each one of the future architecture?

6k views6 Upvotes4 Comments

Head of IT in Software, 5,001 - 10,000 employees
Before we go to the technical requirements and the to be state, I always challenge the team with these questions

1. What do we want to solve here ?
2. What are options do we have?
3. How is the economical impact if we deploy one of that solution ?
4. Do we have the resource to do it?

Once you have those defined, it should be easier to set the goal and boundaries for batch workloads you’re aiming
Software Engineer in Software, 51 - 200 employees
Business Objectives and Use Cases

Analyze the volume and velocity of data that needs to be processed

onsider integrating with data lakes, databases, external APIs, etc

Event-driven architectures (e.g., AWS Lambda, Apache Kafka) are suitable for real-time processing
IT Analyst in Media, 1,001 - 5,000 employees
Look out for single report creating platform across the enterprises. To save data and logs for years to compare the growth and control the value of data. To build on single platform.
Senior IT Analyst - data engineering in Real Estate, 1,001 - 5,000 employees
Batch modernization involves transitioning from traditional batch processing to more modern and efficient architectures. The target future architecture for batch workloads often includes elements like data streaming, event-driven processing, and technologies like AWS Batch or data mesh. When choosing between these, consider factors like workload characteristics (e.g., latency requirements, data volume), scalability needs, cost considerations, and the existing skill set of your team. Data streaming is suitable for real-time, low-latency processing, while event-driven architectures enable efficient handling of event-triggered batch jobs. AWS Batch offers scalability and managed infrastructure for batch workloads, and data mesh is beneficial for handling complex data pipelines. The choice depends on aligning these factors with your specific batch processing requirements.

Content you might like

Improving the developer experience (DX)39%

Improving user/customer experience65%

Solutions to measure and report on code/application quality54%

Consolidating tools to reduce context-switching for your developers28%

Improve Application Performance Monitoring (APM) Capabilities8%


1.7k views3 Upvotes

The future of retail belongs to automated and cloud-based solutions58%

Merchants are not ready34%

Maybe, but not by 20258%


1k views3 Upvotes

Engineer in Services (non-Government), Self-employed
In one of the BUs, the GPT capability of a third-party chatbot was activated for HR Knowledge Base. 

319 views16 Upvotes1 Comment