Running Multimodality locally

Use this page to run either extracted Multimodality project: image generation or image understanding. Both need the same Vite key change and the same OpenAI SDK update. After that, follow the repair for the project you have. That key change is the one every extracted course project needs; Running Chef Claude locally applies it to a React project with two provider routes.

What you need first

Install a supported LTS version of Node.js. Node 24 is recommended and includes npm.

Check that both commands work:

$ node --version
v24.18.0
$ npm --version
11.18.0

You need an OpenAI API key with billing and access to the model you use. Image generation access can require organization verification.

Use a temporary key with a low spending limit. Both projects make paid OpenAI requests from browser code, so anyone who opens the browser's developer tools can read the key.

JunoWhat you need first Install Node.js LTS and set up a temporary OpenAI key with billing and a low spending limit. Image generation can also ask for organization verification, so check that before you start building. I once spent a whole evening on a key with no billing attached, and the error messages never once said so!

JunoWhat you need first Vite can serve the page long before your OpenAI project has billing or model access, so a page that loads proves nothing about the account. The first paid image request is what proves the key works. Keep that key scoped to a low spending limit so a mistake stays cheap.

JunoWhat you need first Local bundling and OpenAI access fail independently, so treat them as separate prerequisites. The browser receives the key, which means every person who opens the page receives it too. Set a low spending limit now, and move the request to a server before anything that resembles a deployment; I have revoked a key after a shorter exposure than this one.

Open and prepare the project

Open a terminal in the extracted folder containing package.json:

$ cd path-to-your-downloaded-project
$ npm install
$ npm install openai@latest

The second install updates the old course SDK before using current image APIs.

Create .env and .gitignore beside package.json:

VITE_OPENAI_API_KEY=your-openai-api-key

.env
node_modules/

The .gitignore entries keep two things out of any repository you create from this folder: your key, and the node_modules folder that npm can rebuild at any time. Ignoring files and good habits covers the wider habit.

In index.js, change the OpenAI client key from:

apiKey: process.env.OPENAI_API_KEY,

to:

apiKey: import.meta.env.VITE_OPENAI_API_KEY,

The key is included in the frontend

This is acceptable only for temporary local learning with a restricted key. Image requests can cost money. Never deploy or share this browser-only version; move the OpenAI request to a backend first.

JunoOpen and prepare the project Open the extracted folder containing package.json, install its packages, then install the current OpenAI SDK on top. Your temporary key goes in .env, and .gitignore keeps that file out of Git. A key that never appears in a commit is a key you never have to replace!

JunoOpen and prepare the project Both projects take the same two edits: the SDK update and the Vite key substitution. Vite reads .env at startup, so restart it after changing the file. And every VITE_ value is sent to the browser, so the prefix marks a value as published rather than hidden.

JunoOpen and prepare the project Upgrade the SDK first: the course version predates the gpt-image request shape, so a new model call through the old package fails with errors that look like mistakes in your own code rather than an outdated package. Only after that, replace the discontinued model ID and the response handling. I once did those two steps in the opposite order and spent an hour auditing code that was never the problem.

Repair the image-generation project

The course snapshot calls the discontinued dall-e-3 model and expects a hosted image URL. Following the current OpenAI image-generation guide, change its image request to use gpt-image-2:

const image = await openai.images.generate({
  model: "gpt-image-2",
  prompt,
  size: "1024x1024",
})

Where the code reads image.data[0].url, replace it with a data URL made from the returned base64 image:

`data:image/png;base64,${image.data[0].b64_json}`

Keep the surrounding assignment or markup from your extracted file.

The base64 string is the entire image inlined into the response, so expect payloads of a megabyte or more. A version with a backend would save the image server-side and send the page a short URL instead of a long data URL.

JunoRepair the image-generation project Point the request at gpt-image-2 and turn the returned base64 value into a PNG data URL. The discontinued model and the old .url field are part of the same change, so update both. If you change one and not the other, the image will be broken, which is exactly the mistake I made first!

JunoRepair the image-generation project Two things changed: the model you request, and the field that carries the image back. Ask for gpt-image-2 and read b64_json instead of .url, keeping the surrounding assignment from your extracted project. If the picture is broken, check the data:image/png;base64, prefix before you change anything else.

JunoRepair the image-generation project Base64 means the whole image is carried inside the JSON response, so expect payloads of a megabyte or more and a data URL far longer than the rest of your markup. A backend version would store the image on the server and send the page a short URL instead. Inlining is acceptable for a local exercise; in production it is a cost you see on the bill.

Repair the vision project

In the Vision Part 2 project, replace the discontinued model ID:

model: "gpt-4-vision-preview",

with the image-capable gpt-4o-mini:

model: "gpt-4o-mini",

The existing message format can still send text and image inputs.

If the updated request fails, send a text-only message to gpt-4o-mini first. A success there confirms your account can use the model, so the image part of the message is what to inspect next.

JunoRepair the vision project Replace the discontinued vision model with gpt-4o-mini and keep the text-and-image message format unchanged. Only the model ID changed, not the shape of the request. Changing fewer lines means fewer lines that can break, a lesson I keep relearning!

JunoRepair the vision project This repair replaces the model and nothing else; the response-shape rework belongs to image generation, not here. Point the request at gpt-4o-mini and keep the multimodal content array unchanged. If the call still fails after that replacement, check account access before you check your own message code.

JunoRepair the vision project Change only the discontinued model ID and preserve the multimodal content array. If the call fails, send a text-only request to gpt-4o-mini first: success there proves account access and points to the image handling as the cause. Narrow down which part of the request fails before you rewrite any of it. I once rewrote it first, and that cost me an afternoon.

Run either project

$ npm start

Open the Local URL printed by Vite. In the image-generation project, submit a prompt and look for the generated image on the page. In the vision project, submit an image and question and look for the model's text response. Restart after changing .env, and stop the project with Ctrl+C.

JunoRun either project Run npm start, open the Local URL Vite prints, and try the flow end to end. A generated image or a vision reply is the success signal, and every attempt is a paid request. I write my test prompt before I start clicking, so I spend less money and waste less time wondering whether to try again!

JunoRun either project Vite starting proves the adapted bundle compiles; it says nothing about the account. One live request checks the key, billing, model access, and the project-specific response path in a single call. Restart Vite after any .env edit, or the old value keeps being served.

JunoRun either project Each project has its own success signal: image generation must receive base64 and render it as a data URL, vision must get its mixed text-and-image message accepted. Check against the right one before you change any code. Every retry is billed, so read the console first and make the second attempt an informed one.

Troubleshooting

process is not defined: Replace the remaining process.env key read in index.js with the import.meta.env.VITE_... form.

The generated image is broken: Confirm the code reads b64_json and adds the data:image/png;base64, prefix instead of reading .url.

The model is unavailable: Confirm billing, organization verification, and model access in the OpenAI project. Provider availability can vary by account.

The SDK reports an error about running in a browser environment: npm install openai@latest keeps the SDK's browser guard, which refuses to run with an API key in browser code. If your extracted project does not already pass dangerouslyAllowBrowser: true in the OpenAI client options, add it beside the apiKey line. The option name is the warning: this stays acceptable only for a temporary, restricted key used locally.

The request works but the page stays unchanged: Check the browser console for a rendering error and compare the returned field with the relevant repair above.

JunoTroubleshooting Check these in order: fix any leftover process.env read, confirm the model, then check b64_json for image generation or the browser Console for a rendering error. Most failures here are one unfinished edit rather than a mystery. Check the ordinary things first; it took me far too long to make that a habit!

JunoTroubleshooting Separate the Vite substitution, account access, response shape, and page rendering; each layer fails on its own. A request can succeed and still render incorrectly, so the Network tab and the page can show different stories. Match the symptom to one layer before you edit anything.

JunoTroubleshooting Read the provider response itself before you change any UI code, because the payload shows what the model returned and the rendered page does not. Keep image-generation output and vision message handling separate; their failures look similar but have different causes. And if the upgraded SDK refuses to run in the browser, that guard is working as intended, so treat dangerouslyAllowBrowser as a local-only override rather than a production answer.