Yesterday I was reading comments on a news article (Note: Do not read comments on a news article). One of the commenters was saying how if only we could reform patent laws, companies around the world could just start making the vaccine right away and supplies would skyrocket. Putting aside the actual supply chain challenges in terms of getting raw materials and components, this isn’t feasible and it’s not feasible for a really great reason.

I remember when I was working in Louisiana and flying back and forth every few months, I saw that a Waffle House had disappeared. A few weeks later when I came back, a new one had been built. On my next trip I had breakfast there. In just a couple of months they had demolished and rebuilt a restaurant and had it up and running. How could they do this? Waffle House is a pretty standard design. You need a counter, some booths, a griddle, a kitchen, washrooms. Design takes very little time, and you need only to construct a simple box, order basic kitchen equipment and furniture and basically by the time your neon “Open” sign arrives, you’re ready to go.

But wait, you need workers. A few cooks / wait staff, trained on how to make eggs, breakfast meat, waffles and coffee, supplemented by training and procedures from corporate. Of course there will be a health department check as well but unless there are issues, this also is fairly painless.

And this is as it should be. For sure, food preparation requires care but we don’t need to worry about keeping it sterile, and generally speaking if the coffee isn’t made quite right, nobody’s going to die. (Depending on how bad it is they might want to, though).

It just so happens I’ve been working in this area of pharma/biotech for about three decades so I can tell you a lot of the moving parts that happen behind the scenes.

After basic design happens and people have a reasonable idea of how big the building has to be and what equipment is needed in it, people need to identify what are called user requirements. These are the basic operating requirements for whatever equipment they’re looking for. What it’s made of, how it operates, sizes, functionality, it is all written down line by line with a level of detail the end user of the equipment would understand. So for a fridge you might say “Fridge must be this size. Freezer must be on top and this size. It must have an ice maker. It must have a fan built in to ensure that the temperature is uniform throughout. The exterior must be stainless steel and easy to clean.” This last one is very commonly seen in our world. Cleanliness is, as you can imagine, very important. Everything must be cleanable to the point where there aren’t even any hard corners in a room. They’re curved so no dirt gets stuck in there. Things like electrical boxes don’t have flat tops, they are tilted so things can’t collect on them. And nearly everything is stainless steel so it stands up to cleaners and doesn’t corrode.

Now these User Requirements are sent to vendors and when they’re purchased (or custom manufactured) specifications are written to a higher level of detail. Now we’re talking about the specifics of how it works. In some cases this is just an owner’s manual but for many pieces of equipment it’s a specific “Functional Specification.” Our fridge user requirements might’ve said “The fridge must be a smart fridge that can notify me by phone if someone forgets to close the door.” but the functional spec says what that looks like. There’s a special app, it has a door sensor that will set off the alarm after a time that the user can set between 5-60 min.

Then after that come the detailed designs. For “off the shelf” equipment like fridges, freezers, scales and the like there may not be a lot of this info but for custom equipment it will be required. This will include detailed design specifications for the software. What are all the configurable settings for and what should they be set to? How is it wired, where does the piping go? What valves are we using on the water system and how does the equipment that uses them tell them to open?

All throughout this process teams from quality, engineering, IT, manufacturing, safety and others are reviewing these specifications. They’re looking for two reasons. One is that they want to make sure they’re what we need. The other thing they’re doing is to ensure that it all traces to one another. The user requirements are called out again in detail in the functional specification or manual. The design documents explain what’s happening in the functional specification.

As these get finished and approved by representatives from the team the design starts to come together. Once it’s progressed enough to get underway, construction starts. And there’s no shortage of specifications here either. Construction has detailed guidelines for how things should be built, how they should be installed and how it should be documented. These design requirements are not negotiable. Here are a few examples:

Systems like high purity water systems that make things like “Water for Injection” have to be built with very specific materials whose identity is not only tested but traceable to a certificate. Every weld in the pipe is numbered and inspected and filmed with a boroscope (like an endoscope for pipes). They’re looking for the slightest imperfection – pitting where something could collect, “rouging” which could turn in to corrosion. Everything must be sloped properly so that no water can stand in a pipe that isn’t always flowing (like near the ‘faucets’ which we call points of use) or on process equipment. But the bulk of the water must always be flowing. Water systems in a pharmaceutical manufacturing plant circulate in “loops” and the flow rate must be high enough to create “turbulent flow” – there should be no eddies where anything can get trapped. Probes must be installed to monitor its quality continuously. It must also be designed to either be always running hot or able to be routinely sanitized.

Water is not the only “Clean utility” here. Compressed air is often used in processing as are other gases like oxygen, nitrogen or sometimes CO2. In all of these cases these must be 0.2 micron filtered meaning that bacteria cannot pass through. The gases must also be of a certified purity and grade.

Facilities, as I said, must have smooth edges. No windowsills, no corners, minimal flat surfaces. There should also be a bare minimum of nooks and crannies where dirt could hide and collect. Some facilities are designed even to be able to be fully washed down while others can be decontaminated with vapourized hydrogen peroxide. Imagine how clean your house would be if it were like this!

• 

• 

Almost all of the work must be done in a cleanroom. The further along in the process (closer to the final filled vaccine vial) you get the cleaner it needs to be. This means that the air must not just be HEPA filtered to remove particles, it must have lots of flow. The air in every room must be changed from 10-20 times to up to 200+ times in every hour. The purpose of this is to ensure that if any particles are generated in the room, the air they’re in will be blasted out and replaced with fresh filtered air. We also design rooms such that air always flows from cleaner areas to areas of lower classification so when you open the door particles don’t go in, they get blown out. In addition, when changing room classifications (cleanliness levels) there should always be an airlock in between. These can be used to bring things in/out or to bring people in/out. These should be dedicated and unidirectional. So you will have a personnel entrance airlock, an exit airlock, and one each for materials as well.

And what about those things that get brought in and out? People have to carefully don clean and often sterile garments. At a minimum in the areas far from the final product that might just be a lab coat or dedicated plant clothes, shoe covers and hairnet / beard cover. In the cleanest of clean rooms where final filling takes place people must wear clothes that have been sterilized. Every bit of skin is covered, often with two sets of sterile gloves. A hood is worn and goggles cover the eyes. A mask is always required. (Note for anti-maskers, this mask must be worn all day and only taken off when leaving the room for break or lunch. This has been the case for decades.) The point of this is that humans are the biggest source of particles and contamination in the environment. Even your cheeks shed skin cells so they must be covered.

• 

• 

Raw materials and components all have specifications and approved vendors whose quality systems have been audited. When they arrive from the vendor they are segregated in the warehouse until they’re tested and released. Then to get them in the clean room they’re either wiped down with alcohol, placed in the airlock and brought in after enough time for the alcohol to do its job. Or for more critical sterile applications a two-door sterilizer is used. In the washing area the goods are prepped, loaded into a sterilizer and then when the cycle is done, a door on the “clean side” can be open to unload it. In most vial filling operations, the vials are washed and placed on a belt in the washing area. The belt carries them to a “sterilization tunnel” where they’re exposed to hundreds of degrees of heat over a long enough stretch (based on the belt speed) that by the time they emerge on the other side (and into the infeed of the filler) they are sterile. Not only are they sterile, they’re “Depyrogenated” meaning that even the fever-producing chemicals that bacteria can leave behind are completely destroyed. Usually the filler infeed will also have its own locally controlled environment or isolator made of clear plastic and chemically sterilizable so that the environment in which open vials are moved, filled and stoppered is completely isolated from the area where the filling operators are standing, monitoring and operating the equipment.

These design innovations are excellent but in the Pharmaceutical world, we don’t blindly trust. We don’t just take the certificate from the vendor saying “I built what you asked.” put it in a file, and start up production. There’s still work to be done. Months or even years, in fact.

As the equipment is being built, other technical staff (of which I’ve been one) start reviewing the specifications for the equipment. We look at the design documents and write up test plans that walk someone through how to verify it was built and supplied as the vendor said it would be. They also need to verify that it’s installed correctly. For a water system, for example, people will write tests to make sure everything matches the drawings, all of the correct instruments and components were delivered and installed in the right places. It’s supplied with the correct water, electricity and other supplies. Tests will be designed to review material certificates, to look at how the pipes are sloped. They must also verify that all instruments are calibrated and wired correctly. In other words, we might have a temperature gauge that reads 0-150°C, but who says that’s right? Someone has to test that – usually at three points, min, middle, and max and verify that it’s close to a verified standard. That standard was also calibrated against another instrument and you can follow that trail all the way back to the National Institute of Standards and Technology.

And we’ve only just begun.

Now we look at how the system operates. Remember that manual or functional specification we had? It’s time to test everything it says it can do. If it were a car we’d be turning every dial, every function, even the ones used by the dealer. We’d test the fog lights, the heater (all fan speeds and the oscillating vents), the A/C, cruise control, trip odometer. And those tests would require that the actual results be documented. No “Check! Cruise control worked.” Instead we’d test it at 30, 60, and 120 km/hr, document start/stop time and variance from the setting and compare that to what it was able to do. If it didn’t work, we’d stop and write it up, document the issue, investigation, root cause analysis and any fixes required and then retest. We are not letting this go until every critical function is tested.

Now remember that User Requirement spec? The one that said our kitchen fridge should be 2-8°C? We’re not going to even take that for granted. You know how your fridge has that spot the veggies always freeze in or the meat spoils in? That’s not going to happen here. We’re going to put a 10-20 temperature probes throughout the fridge and make sure that the temperature is uniform everywhere. We’re going to test it empty and then we’re going to load it up to our maximum. If something doesn’t work right? We’re going to investigate it, figure out the issue and fix it.

We get even more serious when it comes to critical systems that could have an even more direct impact.

Before we can use a room, all of the above testing must be done and then we do several weeks of environmental monitoring, watching not just temperature and humidity but actually going to several parts of every room and taking samples using laser particle counters to make sure that we actually are able to meet the cleanroom requirements. We take other samples that draw air across an agar plate that we send to a microbiology lab to make sure we know we are under control when it comes to microbes. We do it when the rooms are empty and unused and we do it again when people are doing simulated activities to make sure that the activities they’re going to do in there will not generate so many particles the room’s ventilation can’t keep up. This is such an important activity that even after production starts, there will be routine monitoring.

Water systems are sampled for several weeks at every single point of use. Samples are tested for bacteria and also purity. Any issue noted is investigated and addressed. This sampling is most intense prior to startup but also, like the room monitoring, goes on for the lifetime of the facility.

If we’re filling a vial, the filling machine must be incredibly accurate and so before it may be used we verify that it can deliver accurate volumes – within a fraction of a millilitre. And for a high speed filler that could mean that accuracy is maintained even at above 100 vials per minute. (But don’t worry, they’re also measured as a part of their inspection)

Sterilization is such a critical activity that we take no risks at all. Like the refrigerators and freezers, we place many probes in them and run the cycles. First we do it empty. Then we test it with various loads (maximum and minimum) to make sure we’re still meeting the correct temperatures. When we have a load we place probes actually inside the load to make sure the steam is getting to the parts we want to sterilize. And even then we don’t call that good enough. We place small strips of paper inside what we feel are the hardest places for steam to reach. On the strips are Geobacillus stearothermophilus, a thermophile that lives in places like hot springs. During normal life it can live in 75°C water but if it gets hotter it doesn’t die, it puts a hard shell around itself to create a spore that requires prolonged time at over 121.1°C to destroy. We know the exact quantity of spores on each of the strips – around a million each. (Even that’s certified by the vendor). Then we run the cycle. We’ll have temperature data a photo of the load and spore strips inside. They should be killed. But we send them to the micro lab to see if they grow. But we don’t take the vendor’s word completely – we send a control – an untreated spore strip with living spores on it to the lab. That strip should grow and the others should not grow. Until we prove each load three times this way the sterilizer can’t be used.

And even after the sterilization process has been tested (validated), we must go back and periodically test it to verify that they still work.

Imagine this. All of the design considerations in building this, every critical one verified. All of the equipment, hardware, software functions verified. Critical environments and processes not just tested once but at least three times under both static and operating conditions and then periodically over the manufacturing lifetime of the building.

None of this is going to work without procedures. Every function that someone does in a pharmaceutical facility is done per Standard Operation Procedure (SOP). So every performance test that is done means that it must be done as per the procedure you used when testing it. If you need to change the procedure? You have to test again or provide a solid rationale as to why it hasn’t changed anything critical. In every facility there are hundreds of procedures from how to clean the floors to how to fill vials, how to put your sterile gowning supplies on, how to test product or even how to properly document something that you do. If you do it to make vaccines, it’s done following a procedure. If it’s generating data or is related to the batch, you’re signing something. For every process there’s a batch record that walks everyone step by step through the process. Imagine a step by step recipe in which you sign and date each step. Now imagine that in addition to just following the directions you’re recording how much flour you use (to 0.1 gram), the lot number of the salt, the amount of time you mixed and the speed the mixer was set to. The oven (that was verified to be uniform in temperature (not like my home oven!) was set and the person who did it documented it, the person who loaded it documented it along with the start time. The person who took it out of the oven also documented the end time and signed.

But you can’t even sign those until you’re trained. For simple procedures it might just be reading the procedure and signing that you read it. Others may have training sessions with quizzes or on the job testing. All of that is documented and filed away – often for literally decades.

All of this testing has to be done. All of the procedures must be written, everyone must have been trained on all of the procedures they’ve been trained on. And believe it or not we’re still not done. We don’t know that the manufacturing process will work. We haven’t proven that after we’ve manufactured our product and dirtied all of our equipment that we can clean it afterward.

Process validation requires several lots of the product be made and tested for consistency, yield and quality. To go back to our kitchen, we now have a recipe for bread that we are happy with. We have procedures and we’ve been trained. We make the bread three times using the same recipe. We may vary some things within the acceptable range. Sometimes our water might be a little warmer or colder. At the end the bread is sampled and verified not just to be as good every time but to effectively be identical.

Cleaning works the same way. We can’t just throw a dish in the dishwasher and say “Well, the dishwasher cycle worked so the dish must be clean.” Those of us with dishwashers know how wrong that is. So we need a cleaning procedure to start. Do we pre-rinse? How long? Hot water or cold? Any scrubbing? How much? Then what detergent do we use. No, we can’t just pick what’s on sale. Pick one. Fine, Calgon it is. If you prove cleaning with Calgon you’ll have to use it until you can prove another one works just as good.

Now we have to dirty our dishes – worst case. We’ll still follow our procedure but maybe we’ll even run it on “light load”. At the end we’ll check to see if it’s clean. In our world that means sampling. When we verify the cleaning of a tank, for example we must first verify it’s been properly dirtied then run the cleaning cycle. Then, someone will even go inside the tank and swab various areas of the wall, inside pipes or in hard to reach parts. Those swabs are sent for analysis to make sure there’s nothing on them – detergent or dirt. And don’t worry, that test for dirt or detergent? Even that’s been proven to work right down to quantifying just how much we can extract from the swab. We also verify rinse samples as well as it is finishing to make sure that that rinse water is clean. Again, no dirt or detergent.

This is just a tiny fraction of what’s done, there are volumes and volumes of data. And our hypothetical facility still can’t start production without a pre-approval inspection from the FDA and other regulatory bodies (depending on the country of sale). It’s not enough that people like me in the Quality group think a good job was done in testing, the government also comes in and reviews the data, walks through the manufacturing facility, and asks questions. They don’t just ask questions to people in charge, they walk the floor and can come up to any worker and ask them about their job and what they do to make sure they’re properly trained. They may even watch some of the manufacturing in progress, following along with the procedure to see if it is being done as they said it would be.

And imagine – every facility where every medicine you’ve ever taken was made has gone through this – and this is only a really high level summary. And now you can see why people aren’t just going to quickly get a COVID-19 vaccine manufacturing facility built and running.

But now I can already hear some of you asking. If that’s the case, how did those already manufacturing do it?

I don’t know the specifics about any of the vendors but there are some likely answers.

One is that some have used contract manufacturers. These facilities are designed and the testing not related to the product is already done. The clean rooms are there, the equipment is there and the people are trained and qualified. All that is required is to prepare product-specific instructions, train on them and do the cleaning and process testing. These facilities have lots of flexibility because their job is to manufacture for others with whatever process is sent to them.

Another option is that a company repurposed an existing facility. They would have to stop production of whatever was in there, test and document that they’ve fully removed all traces of that product, then treat it in the same way as the first option. The procedures for things like room cleaning and water system maintenance are in place and people are trained. You just need to write procedures for and test the production and cleaning process for the new vaccine.

If you had a facility nearly ready for another project you could also decide on the fly to, for example, send that original product out to a contract manufacturer and do the COVID vaccine production in-house.

So there are a number of options, and as I’m not working with any of the current manufacturers now I can’t say how they did it. And really, if I were working for one of them, most likely I still couldn’t tell you due to client confidentiality.

And finally, it is important to recognize that these test requirements don’t stop at the factory. The logistics company has to demonstrate that their software properly tracks the lots of drugs going places and can, at any moment, tell you what lot went where in the event of a recall.

When those vials are removed from the freezers at the manufacturer (the ones we tested by monitoring a couple dozen points in them for 24 or more hours with and without load, testing the response when the door was opened and how long it could last with a power failure), they’re loaded in to shipping coolers. The makeup of those shippers down to the number of gel packs and where the boxes are placed was tested. The maintenance of temperature of the product? That was also tested over the entire route even in the heat of summer (as vaccines are mostly water, this didn’t have to be product-specific). The intermediate warehouses from contract logistics companies? Their freezers and freezer monitoring systems were also tested to the same level as the manufacturer. (The manufacturer even audited them to check out their procedures, testing, training and quality systems.). And then, even when they arrive in your local drug store. That freezer also required testing and monitoring.

Another thing I’ve heard folks worrying about is some of the manufacturing issues recently found. For example, those by one of the contract manufacturers at J&J. This vendor had started manufacturing before their pre-approval inspection hoping that once the FDA said they could begin production they could also release the lots that had been produced. However there were some manufacturing errors resulting in contamination and later the inspection by the FDA turned up some serious issues. This is serious and concerning but also shows that the system is working as it appears the quality unit at the manufacturer caught the issue by reviewing documentation and lab results. It isn’t common for errors like this to be made, but it is for this sort of reason quality units exist, and are required by law to be independent of manufacturing units. And of course, having an FDA review of the facility ensures that the full scope of the issue is investigated.

As you can see from all of this, there are a number of checks and balances in place, procedures to catch not just errors that machinery make but errors made by individuals. Oversight by an internal and independent quality organization keeps things in check as does the fact that in order to start and remain in production, routine inspections are required by the regulatory bodies of every country to which a manufacturer ships their product.

So when you start to worry “Who’s making this stuff?” and “Are they really concerned about quality?” know that there are literally hundreds of people just like me working from the chemical plants where some of the ingredients are made, to the vaccine manufacturer, the warehouse and the end user. We’re all looking out for quality because it’s not just going in you, it’s going in us, our families, our friends and loved ones. We take our job incredibly seriously. And I’m confident enough that as I sat in the chair getting my first vaccine a couple of weeks ago, there was not a moment’s hesitation. I may not have reviewed the data that supported the release of the facility or specific lot of vaccine, I know what went in to it and have no concerns. I hope, having read this series you, also have a few less concerns.