Turning process insight into tangible gains
Organizations in many industries, such as manufacturing, face a common challenge: improving yield. Whether the goal is more product, fewer defects, or higher efficiency, success often depends on refining a complex, multi-step system.
In this article, we will explore data-driven methodologies for optimizing complex manufacturing systems and maximizing yield. By combining process insight with machine learning techniques, manufacturers can continuously refine operations, reduce waste, and drive measurable improvements.
Taking a repeatable approach to process optimization
While every system is unique, the path to improving a process is repeatable and applied across the solution hierarchy:
- Define the process and determine points of evaluation.
- Identify known and unknown variables.
- Transform the known variables to model the unknown ones.
- Simulate and optimize with a loss function.
Following this path will pinpoint the areas where errors are occurring, enabling refinement of the process. Once the locations of losses are identified, one can propose a hypothesis to address each specific sub-problem in the system. This approach follows the typical scientific process within the development of data solutions.
We will walk through a simplified but practical example and show how this process can be implemented.
Applying the methodology to manufacturing
In this example, let’s take a goods manufacturer looking to maximize the yield of a finished product using a fixed amount of input materials. In this case, yield could be defined simply as:
Let’s assume the input mass is relatively fixed, therefore, optimization is about minimizing waste and maximizing usable output through better control of the process. A typical production process can be broken into sequential stages.
- Preprocessing: Prepare raw inputs (e.g., melting, grinding, cleaning).
- Formulation: Combine inputs.
- Processing: Heat, press, and apply transformations to the inputs.
- Packaging: Finalize and package the final product.
Each stage has controllable parameters (temperature, time, input ratios, pressure, etc.) and measurable outputs (moisture, shrinkage, density, defects, etc.). At each step in the process, the manufacturer should weigh or evaluate yield loss as it allows engineers to gather data and evaluate how well a set of parameters performs at increasing production.
Preprocessing optimization with machine learning
While an overall yield prediction model can support quality assurance and continuous improvement, it is often more effective to break the process into stages and develop sub-models for each. These models help improve individual steps in the manufacturing workflow.
Take food production as an example: if yield inconsistencies are due to moisture and fat loss in meat, it is reasonable to hypothesize that factors such as meat temperature, grind size, water content, fat ratio, and cooking profile influence final loss. Sub-model development for preprocessing requires the following steps:
- Collect real world batch data: initial meat composition, grind size, water ratio, cook times/temps, final weights.
- Train a regression model to predict expected yield loss from input features.
- Use the model to recommend optimal input parameters (e.g., fat ratio, cook ramp) to minimize loss per batch.
- Apply and validate input parameters through A/B testing.
Each sub-model can be a traditional regression algorithm, or a more complex neural network trained on historical batch data. In any case, machine learning enables the system to learn from prior behavior and offer intelligent, data-driven predictions.
A modular approach not only simplifies development but also enables simulation of different production scenarios. These simulations can guide process optimization while reducing the number of real-world experiments required.
As a starting point, let us assume we have a dataset structured as follows. Our goal is to predict the product weight after the preprocessing step (e.g., defrosting and grinding), which can then be used to calculate yield loss:
The following code can be used to generate a sample dataset.
import pandas as pd
import numpy as np
import random
# Set seed for reproducibility
np.random.seed(42)
n_samples = 10000
# Create synthetic preprocessing data
df = pd.DataFrame({
'batch_id': [f"{i+1}" for i in range(n_samples)],
'initial_weight_kg': np.round(np.random.uniform(10, 25, n_samples), 2),
'meat_temp_c': np.round(np.random.uniform(-4, 5, n_samples), 1), # post-defrost temp
'grind_size_mm': np.random.choice([3, 5, 8], n_samples), # grinder options
'water_ratio': np.round(np.random.uniform(0.05, 0.2, n_samples), 2),
'fat_ratio': np.round(np.random.uniform(0.1, 0.3, n_samples), 2)
})
# Simulate yield loss ratio (pre-cook)
def simulate_weight_loss(row):
# Assume baseline moisture/fat loss during grinding/mixing
base_loss = 0.08
temp_effect = -0.002 * row['meat_temp_c'] # colder meat = less loss
grind_effect = 0.005 * (row['grind_size_mm'] - 3) # finer grind = less loss
water_effect = -0.05 * row['water_ratio'] # water added reduces % loss
fat_effect = 0.1 * row['fat_ratio'] # more fat = higher loss
total_loss_ratio = base_loss + temp_effect + grind_effect + water_effect + fat_effect
total_loss_ratio = np.clip(total_loss_ratio, 0.02, 0.15) * random.uniform(0.5, 1.5)
return np.round(row['initial_weight_kg'] * (1 - total_loss_ratio), 2)
df['final_weight_kg'] = df.apply(simulate_weight_loss, axis=1)
# Optional: add calculated yield loss for reference
df['yield_loss_ratio'] = np.round(1 - (df['final_weight_kg'] / df['initial_weight_kg']), 3)
print(df.head())
df.to_csv('YieldOptimization/data/preprocessing_data.csv', header=True)
Frozen meat is expected to weigh less after processing due to water loss. Minimizing loss includes reducing meat left in machinery or through trimming. Once we have our data collected and available, engineers can train a regression model using historical data with the following Python code.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
import matplotlib.pyplot as plt
import seaborn as sns
import joblib
df = pd.read_csv('YieldOptimization/data/preprocessing_data.csv')
# Step 2: Prepare data for ML
features = ['initial_weight_kg', 'meat_temp_c', 'grind_size_mm', 'water_ratio', 'fat_ratio']
target = 'final_weight_kg'
X = df[features]
y = df[target]
# Step 3: Train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Step 4: Train a regression model
model = LinearRegression()
model.fit(X_train, y_train)
# Save the trained model
joblib.dump(model, 'YieldOptimization/data/yield_predictor_model.pkl')
# Step 5: Evaluate
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
rmse = mse**0.5
r2 = r2_score(y_test, y_pred)
print(f"RMSE: {rmse:.2f} kg")
print(f"R² Score: {r2:.3f}")
# Step 6: Plot results
plt.figure(figsize=(6, 6))
sns.scatterplot(x=y_test, y=y_pred)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], '--', color='gray')
plt.xlabel("Actual Final Weight (kg)")
plt.ylabel("Predicted Final Weight (kg)")
plt.title("Model Performance: Actual vs Predicted")
plt.grid(True)
plt.tight_layout()
plt.show()
The model developed during this phase constitutes a mathematical representation of yield subsequent to preprocessing. This model enables us to forecast yield based on a predetermined set of input variables.
Discovering the optimal parameters
With our machine learning model, we can feed it different scenarios using the following Python code. This code shows how to input various values to forecast the yield.
import numpy as np
import pandas as pd
import joblib
model = joblib.load('YieldOptimization/data/yield_predictor_model.pkl')
# Sample data (each row = one batch)
sample_data = pd.DataFrame([
{
"initial_weight_kg": 18.0,
"meat_temp_c": 2.0,
"grind_size_mm": 5,
"water_ratio": 0.12,
"fat_ratio": 0.18,
},
{
"initial_weight_kg": 20.0,
"meat_temp_c": 0.0,
"grind_size_mm": 3,
"water_ratio": 0.10,
"fat_ratio": 0.22,
}
])
# Predict final weights
sample_data["predicted_final_weight_kg"] = model.predict(sample_data)
# Compute yield loss
sample_data["yield_loss_kg"] = sample_data["initial_weight_kg"] - sample_data["predicted_final_weight_kg"]
sample_data["yield_loss_pct"] = (sample_data["yield_loss_kg"] / sample_data["initial_weight_kg"]) * 100
# Display results
print(sample_data)
To determine which parameters to test in real-world scenarios, engineers can perform an optimization step using the following code to evaluate inputs for optimal loss function outcomes.
from scipy.optimize import minimize
import numpy as np
import joblib
# Load the pre-trained model
model = joblib.load('YieldOptimization/data/yield_predictor_model.pkl')
# Define the features
features = ['initial_weight_kg', 'meat_temp_c', 'grind_size_mm', 'water_ratio', 'fat_ratio']
# Objective: minimize the yield loss (initial - predicted final weight)
def objective(x, model):
# x = [initial_weight, meat_temp, grind_size, water_ratio, fat_ratio]
input_features = np.array(x).reshape(1, -1)
predicted_final_weight = model.predict(input_features)[0]
initial_weight = x[0]
yield_loss = initial_weight - predicted_final_weight
return yield_loss
# Bounds and reasonable starting guess
bounds = [
(10, 25), # initial_weight_kg
(-4, 5), # meat_temp_c
(3, 8), # grind_size_mm
(0.05, 0.2), # water_ratio
(0.1, 0.3), # fat_ratio
]
# Starting guess (initial guess for optimization)
x0 = [18, 0, 5, 0.12, 0.18] # example initial guess
# Optimize
result = minimize(objective, x0, args=(model,), bounds=bounds, method='L-BFGS-B')
# Get the optimal inputs found by the optimizer
optimal_inputs = result.x
# Round grind_size_mm to the closest allowed value
grind_size_mm_options = [3, 5, 8]
optimal_inputs[2] = min(grind_size_mm_options, key=lambda v: abs(v - optimal_inputs[2]))
# Predict the final weight using the optimal inputs
predicted_weight = model.predict(np.array(optimal_inputs).reshape(1, -1))[0]
# Calculate yield loss
initial_weight = optimal_inputs[0]
yield_loss = initial_weight - predicted_weight
# Print results
print("Optimal Inputs:")
for name, val in zip(features, optimal_inputs):
print(f" {name}: {val:.3f}")
print(f"\nPredicted Final Weight: {predicted_weight:.2f} kg")
print(f"Yield Loss: {yield_loss:.2f} kg ({(yield_loss/initial_weight)*100:.2f}%)")
Using the system model, we simulate input parameters to predict yield under different conditions. We aim to identify settings that operators can adjust for real-world improvements. This step recommends input settings for a test group to validate results.
Real-world deployment and iteration
Deploying machine learning in a manufacturing setting is not a one-time effort. Rather, it is an iterative process grounded in experimentation and real-world feedback. Once an initial model is trained, manufacturers should apply the recommended settings on a pilot production line. This controlled environment enables safe testing of optimizations and captures new data under realistic operating conditions.
As new data is collected from ongoing batches, the model can be retrained and refined, creating a feedback loop where predictions become more accurate over time. This iterative process allows the system to adapt to real-world variability, which is something static rules-based systems struggle with.
While the maximum achievable yield may be known in some cases, it is often unclear, which is why simulation powered by machine learning is so valuable. Modeling each production stage separately allows for virtual exploration of the yield landscape and helps uncover how close your current process is to the theoretical optimum.
Over time, sub-models for each step (e.g., defrosting, grinding, cooking) can be stitched together into a higher-level system model. This “parent model” can weigh trade-offs between stages. For example, accepting slightly higher yield loss in one step if it prevents greater losses downstream. This holistic view transforms isolated optimizations into a coordinated, intelligent manufacturing system.
The goal is not just to provide predictions, but prescriptive guidance and intelligence. Experimentation guides operators toward the best possible input parameters at each stage of the process. The result is a self-improving system that combines data, domain expertise, and intelligent modeling to continuously drive better outcomes.
Why this approach works
This methodology for system optimization is powerful because it is:
- Modular: A perfect end-to-end model from day one is not required. Instead, this approach emphasizes optimizing each stage of the process independently, allowing teams to focus on solving smaller, manageable problems first.
- Data and AI-driven: By leveraging historical production data, engineers and scientists can apply advanced machine learning techniques to uncover insights and drive improvements.
- Incremental: Progress is achieved in phases. Teams can deliver early wins within a shorter period, and each model component can be updated or improved independently as new data becomes available.
Many organizations begin by modeling one or two critical process stages, quickly realizing measurable gains. Over time, this modular and iterative approach scales to a full system-level optimization framework. The result: higher yields, reduced waste, and greater profitability.
All code is available in this GitHub repository
Optimizing complex systems is a journey, not a one-time project. If you would like to discuss ways to start building a smarter process today, get in touch. We are happy to help.
