How to start automating engineering processes like a pro

Automation that makes a difference

If you're streamlining your workflow with tools like spreadsheets (e.g. Excel), visual programming environments (e.g. Grasshopper, Dynamo), or basic scripting (e.g. Jupyter Notebooks), you're already doing a great job automating your process. However, I know from first-hand experience the love-hate relationship most engineers have with spreadsheets and tools alike. While Excel, Grasshopper, Dynamo (etc.) are all powerful tools, they have their limitations as well. Think about issues when it comes to scaling, lack of version control, challenges with collaboration, proneness to error, complications with data integrity and security, and so on.

In this blog post, I will explain how you can bring your tools to the next level, so your work becomes more fun, and you will become more scalable, efficient and productive than ever!

Five levels of automation

Before I dive deeper into the practicalities, there's a little piece of theory I want to share first. It's about a framework, called ‘The five levels of automation'. This framework describes the complexity of an automation project and can used to determine the skills you require for a specific development.

• Level 1: Out of the box: You use standard, already existing tools that help you automate part of your process. For example, using a tool to convert a 3D model into a FEM model or using a no-code generator to create a user-friendly interface for your tool.

• Level 2: Simple automation: You automate a part of your process by creating a custom tool or script in Excel, Python, Grasshopper, Dynamo and others. For example, a script to generate a 3D model or drawing automatically based on some input parameters, or a sheet to post-process the results of an analysis.

• Level 3: Integration: You make tools that integrate and exchange data between at least two software packages by using their API or other methods. For example, a script that runs a calculation on a commercial software package and automatically imports the results into an Excel sheet where you perform other calculations or checks.

• Level 4: Multi-disciplinary: The tools you make cross the boundaries of your own discipline and could integrate several commercial programs to automate a larger part of the workflow. For example, a Python script runs both a structural calculation and a geotechnical calculation on two different packages.

• Level 5: Hyper-automation & AI: Besides connecting different software packages, you have advanced optimization routines 10 or use artificial intelligence to compare hundreds of designs and get the best solution across a large part of the design process. For example, the previously mentioned Python script not only runs the structural and geotechnical calculations but also analyses multiple designs until finding a solution that fulfils the requirements on both sides, whilst also producing the engineering reports and drawings.

Resources to start automating like a pro

Out of the box: Creating a UI with no-code

As I mentioned at the start of this blog, many engineers already have automation tools like Excel sheets and visual programming environments. The easiest way to level up these tools is by building a user-friendly interface on top of it.

Out-of-the-box resources can help you get started with this more easily, the result being a web app that you and your colleagues can use together.

Examples of out-of-the-box tools you can use to create a UI for your tools are:

• PyQt: A Python library for creating cross-platform GUIs using Qt. You can design your UIs with Qt Designer and integrate them in Python code for a highly customizable interface.

• Tkinter: Python's standard GUI toolkit, included with most Python installations. It allows you to create simple and functional UIs using a straightforward approach with widgets like buttons, labels, and entry fields.

• Streamlit: An open-source Python library for creating web apps specifically for data science and machine learning projects. Write Python scripts to generate interactive web apps.

• VIKTOR: The VIKTOR platform is specifically created for engineers who want to automate their processes through web apps. With the VIKTOR Visual Builder, you can generate the GUI for your app without writing any code. Or use the Excel Converter to automatically turn your Excel sheets into a web app.

Simple automation: Learning basic Python skills

In my opinion, your real automation journey starts here. In most cases, you want to start automating because there's a specific problem you are trying to solve. You only need to leverage the right technologies when doing so.

From my own experience, Python is one of the easiest ways to accomplish automation. A solid starting point of your journey is enrolling in a Python course which helps you understand the basic concepts of coding and enables you to work with existing code.

• Udemy: Here you can find numerous Python courses tailored to different levels, from beginner to advanced. Courses include practical projects, helping you apply Python to real-world scenarios.

• LinkedIn Learning: Find courses taught by industry professionals, ensuring high-quality content. Upon finishing a course, you even receive a certificate which you can add to your LinkedIn profile.

Following a course enables you to integrate and modify existing code snippets to build custom solutions for your specific problem. Additionally, by engaging with a learning community you will also find valuable examples and more areas to automate in.

• StackOverflow: Q&A site where you can ask specific questions and find answers from experienced developers. It's an excellent resource for troubleshooting and learning from common issues others face.

• Real Python: This platform offers tutorials, articles, and community discussions that help deepen your understanding of Python.

• Reddit (r/learnpython): This subreddit is dedicated to Python learners, where you can share progress, ask questions, and get advice.

• GitHub: Here you can find open-source projects and join in discussions in repository issues, learn best practices, and see how Python is used in real projects.

Places that I like to use to find information and help for the use of specific Python packages are the forums of said packages. For example, if you want to visualize a 3D geometry using Matplotlib.

Last but not least (actually, most importantly if you ask me!), don't be scared to use other resources like Large Language Models (LLMs) via your search engine that can help you generate code snippets using AI.

• ChatGPT: is great at creating code snippets for you!

• Copilot: It's possible your organization has ChatGPT blocked. Luckily, there are some alternatives like Copilot, which is designed by GitHub to assist you with coding by providing context-aware code suggestions. Have Edge installed on your device? Then you should be able to use Copilot.

• Perplexity: This LLM focuses on answering questions by aggregating information from various sources to provide more concise and accurate answers to your queries, making answers very reliable and transparent.

• Gemini: Leverages Google's search capabilities and advanced natural language processing to deliver accurate, contextually relevant responses.

• HACKTOR: This is the VIKTOR platform's very own user-friendly AI-powered app code builder which you can use to generate the code for your VIKTOR app.

Please note that an LLM is not guaranteed to give you perfect code, but it does work extremely well in generating code for well-documented topics such as NumPy and can even teach you how to use plotting libraries as they often explain the code. As plotting is very common in engineering disciplines, I highly recommend trying to generate some beautiful Plotly graphs and asking it some questions to become familiar with it!

Integration: Connecting external software through APIs

Once you have a solid grasp of Python and coding fundamentals, it's time to explore how APIs can connect you to external software, taking your automation capabilities to the next level!

First, let me explain what an API is: an API is a set of rules and protocols that allow different software systems to communicate and exchange data. They enable your application to interact with several services or systems. Automations using an API are all about making a tedious and repetitive process in external tools fast and robust. A simple example could be automating your Excel calculations through a web app.

The easiest way to find information on building your own integration with a specific software, is to Google the name of said software in combination with ‘API’. This will likely lead you to some examples of how this is done. However, building integrations yourself using APIs is quite complex and time-consuming. Instead, I recommend using a platform that offers pre-built integrations, like:

• ShapeDiver: Offers pre-built integrations that you can use to turn your Rhino and Grasshopper files into interactive 3D models, accessible from your browser.

• VIKTOR: Offers many dedicated workers that can help you easily connect to specific software (like Grasshopper, Dynamo, Excel, PLAXIS, ETABS, Revit, SCIA, and more), as well as a generic worker which you can use to run scripts and actions on other machines. This means your team does not require to have the software itself installed on their own device.

Whether you want to use a pre-built integration or built one yourself, these are some useful resources I recommend:

• How to use a Public API
• Integrate engineering and design software in the cloud with VIKTOR

Multi-disciplinary automation: Beyond your own tools

In most cases of automation, you are concerned with improving your own workflow. However, there are also instances in which you rely on input from other stakeholders. Therefore, it can be interesting to integrate the workflows of other disciplines with your own in certain cases.

There are several ways this can be done. This integration can be achieved in several ways. One option is to build a tool for a colleague that connects with yours via an API. Another approach is to create a unified tool that integrates all processes and external software. Regardless of the method, data standardization is crucial for success.

Different applications and software may handle data differently, leading to potential issues like data loss or incompatibility. For example, if a geotechnical engineer and a structural engineer are working on the same project but use different data formats, collaboration can be challenging. The geotechnical engineer might conduct soil calculations, while the structural engineer performs structural analysis. If their data formats don’t match, or if there are differences in units of measurement, the structural analysis could be flawed.

To automate and integrate their work effectively, it's essential to use standardized data formats and protocols. This ensures that data is correctly interpreted and transferred, leading to smoother collaboration and a more accurate project outcome.

Hyper-automation and AI: Optimizing results

In principle, you can optimize your automation process at any stage in your journey, but success depends on having the right knowledge, expertise, and most importantly, data.

I recommend starting with a simple approach. For example, if you’re working with a basic calculation involving two input values and one result, you can create an optimization model to answer questions like: “What is the maximum C-value given specific A and B values?” You can do this using tools you already have, like Excel.

Once you’ve established a standardized process and gathered sufficient data, you can explore AI options. Training your own algorithms with your data can help further optimize processes, though developing AI can be complex and time-consuming, with no guarantee of success.

Some examples of AI models are:

• Supervised Models: For instance, Support Vector Machines (SVM) are useful for binary classification tasks.

• Deep Learning Models: Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks have shown success in infrastructure monitoring and maintenance.

When using AI for optimization, it's crucial to have a consistent process for data collection to ensure accuracy, robustness, and reliability, while also minimizing overfitting. Also, remember these examples are very conceptual and due to their nature, extensive verification and validation of results will be needed to ensure their effectiveness.

Conclusion

I hope this blog and the resources I shared have proven useful in your engineering automation journey. Automation has really opened a world of possibilities to me, and I hope it helps you become more efficient, productive and scalable as well – and (perhaps most importantly) makes your workday more enjoyable every day😉.

If you have any questions about all the things I've shared or if you'd like to have a chat (about engineering automation, or anything else) hit me up via LinkedIn or send me an e-mail at abergman@viktor.ai