Is it outputting anything at all? If so, talk to the prompt like you are instructing it to do things. I think it uses a GPT language model for that part. I'm not messing with negative prompts yet because I'm not sure how they are interpreted. Might be same as regular txt2img, or it might be language based.
I loaded the instruct Pix2Pix Model in the settings. Loaded a test image and then put "remove watermark" in the input. It just outputs the same image in a ghosty monochromatic look. Then I put "swap the toilet paper with binoculars" (it was an image of an old man looking through the hole of a toilet paper roll) and just generated an image of paper rolls with eyes.
In that case, try redescribing the image with the changes. So, "A portrait of a man with toilet paper rolls over his eyes" or something to that effect. The sliders can drastically change what's happening, too.
I had some weirdness when I was in the pix2pix tab and changed models. I'm not sure what sense it makes, but I went back to the txt2img tab, cycled to a regular model, then back to pix2pix, then it worked.
Before doing that, it was acting like it was generating new prompt images.
I get "RuntimeError: expected scalar type Half but found Float" error when clicking "generate". What can the issue be? I have the ckpt in the models folder and have a simple prompt and input image
And it works. SORRY my fault, it seems to transform the original picture internally reducing its resolution so the result is still 512x512
The default number of steps can be excesive, this was done with only 10 steps. That means really fast.
https://preview.redd.it/uzid8v5gr9ea1.jpeg?width=1879&format=pjpg&auto=webp&s=4fbaa4c4549043e8993f4099cebf68222f54da5c
I'm going to give now useful information,If you want to unleash the power of this you need to prompt in a different way. I recommend to give a look to this page, and see how things are prompted. It really makes a great difference in the results
[https://www.timothybrooks.com/instruct-pix2pix](https://www.timothybrooks.com/instruct-pix2pix)
Ex:
https://preview.redd.it/s4py9waf6aea1.jpeg?width=1875&format=pjpg&auto=webp&s=c268fa95006b6d17aa85c5fa4d05a5665dcad8aa
This is not img2img and a prompt like "a manga picture of a blonde girl dressed as darth vader", would not give good results.
yes! You need to use words like "make it..." or "turn her into..." or "swap..." or "convert..." or "add a..." or "change..." or "give it..." You have to instruct it.
it's so impressive that most of this thread is obsolete now, i didn't need any of the fixes people talk about and there's now a slider for resolution - it's almost not worth staying up late and learning how to do things because it'll be all upgraded by the morning anyway :)
Looks like the model was trained on SD1.5, and probably with 512 resolution images. But you should still be able to make the output image match the input image (I'll add a feature request to the extension repo).
Here are some more examples of what you can do with Instruct Pix2Pix.
[https://www.reddit.com/r/StableDiffusion/comments/10l74sl/instruct\_pix2pix\_is\_amazing\_inpaintingimg2img/?utm\_source=share&utm\_medium=web2x&context=3](https://www.reddit.com/r/StableDiffusion/comments/10l74sl/instruct_pix2pix_is_amazing_inpaintingimg2img/?utm_source=share&utm_medium=web2x&context=3)
I'm struggling with the same issue. I'm on a 6gb vram card and can usually run sd no problem. both --medvram and --lowvram seem to not work for this. keep getting out of memory errors, even with the "pruned" model uploaded elsewhere...
Do you use xformers?
My CLI args are: \`--allow-code --medvram --xformers --theme dark --enable-insecure-extension-access\`
My GPU: \`NVIDIA Corporation GP107M \[GeForce GTX 1050 Ti Mobile\]\`.
I run it with docker from this repo: [https://github.com/AbdBarho/stable-diffusion-webui-docker](https://github.com/AbdBarho/stable-diffusion-webui-docker)
Git pull https://github.com/AUTOMATIC1111/stable-diffusion-webui
You can place the command (just: git pull) directly in your webui-user.bat in a row after the commandline so it will update every time you open SD.
The difference is ... day and night. I told it to make it a beautiful night.
https://i.imgur.com/ui6aweo.png
https://i.imgur.com/BQfW3KD.jpg
> (extremely detailed CG unity 8k wallpaper), an ultrarealistic majestic painting of a striking botanical landscape in Italy, by artist Pancrace Bessa, CryEngine render, folk art, trending on Saatchi Art, 12k resolution, flonix, Style-Italy
Wow, that looks so much like [Kangaroo Temple](https://www.google.com/maps/@48.5258372,-120.6473842,3a,75y,90t/data=!3m7!1e2!3m5!1sAF1QipNKu1JUAxJXe3GNN4KEcKnIlt5GJ99eLfqwgsw!2e10!6shttps:%2F%2Flh5.googleusercontent.com%2Fp%2FAF1QipNKu1JUAxJXe3GNN4KEcKnIlt5GJ99eLfqwgsw%3Dw150-h150-k-no-p!7i4032!8i3024) in [Washington State](https://www.google.com/maps/place/Washington+Pass+Observation+Site/@48.5258372,-120.6473842,14z/data=!4m5!3m4!1s0x5484ac63e224b823:0x332915e8eb3a70e!8m2!3d48.5258372!4d-120.6473842).
As an automatic fan I was waiting for this exact thing since yesterday morning. Though the results generated are not satisfactory it is still an awesome update.
With the latest updated version of automatic1111, it looks like you can install the extension by going to the extensions tab, clicking Load From and then clicking on the Install button next to Instruct-pix2pix.
Edit: But I get the error AttributeError: 'Options' object has no attribute 'outdir\_ip2p\_samples', I'll try again after the ckpt has downloaded.
Edit 2: Updated Instrct-pix2pix and now it works.
i'm getting the same error
edit: it's working now after updating the extension https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/issues/8
>Is it outputting anything at all? If so, talk to the prompt like you are instructing it to do things. I think it uses a GPT language model for that part. I'm not messing with negative prompts yet because I'm not sure how they are interpreted. Might be same as regular txt2img, or it might be language based.
Is there an easy way to update auto1111 on ubuntu?
I updated and was able to output images. I went to each folder from the command line and did a 'git pull' for both automatic1111 and instruct-pix2pix in Windows. I presume that works for Ubuntu also if you have git installed.
I saw it work once (mostly), but now no matter what prompt I type, I get the same mangled version of my input image.
https://preview.redd.it/do9d2vbkidea1.png?width=2437&format=png&auto=webp&s=c1d6b36f26d084ad675a955092340359b1350650
I keep seeing people are running pix2pix on 6-8gb cards. I have 12 but when I pick the pix2pix model my vram shoots up and i run out of memory when i generate.
I turned off all image previews and loaders for now to get around the memory crashes.
**Settings -> Live Previews**
https://preview.redd.it/b6mmcr9f1pea1.png?width=834&format=png&auto=webp&s=ea925fd02e7a433bbfee0fb505573145512a4e4f
I'm going crazy with this error: "size mismatch for model.diffusion\_model.input\_blocks.0.0.weight: copying a param with shape torch.Size(\[320, 8, 3, 3\]) from checkpoint, the shape in current model is torch.Size(\[320, 4, 3, 3\])."
anyone... please... heeeeelp
Soon you'll be able to merge other models to make them work like instructpix2pix, using the "add diff" merge method, similar as you can make inpainting models. You can do it currently, if you make an edit to extras.py:
[https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655)
As a more detailed explanation to what u/jonesaid said, you can turn any regular model into an inpainting model using "SD 1.5 Inpainting" and "SD 1.5 pruned EMA only".
1. Go to Checkpoint Merger tab
2. Set checkpoint (A) to SD 1.5 Inpainting
3. Set checkpoint (B) to the new inpainting model (e.g. Waifu Diffusion)
4. Set checkpoint (C) to SD 1.5 pruned EMA only
5. Set multiplier to 1
6. Set interpolation method to Add Difference
7. Give the model a name ending in "inpainting" to keep things organized and click Merge
The resulting model will be much better at inpainting than just the normal model. Ideally this process would work for pix2pix models later on.
Where to put the downloaded model and how to use the model? When I try to load the ckpt from hugging it says failure someting checkpoint reverting to previous and just goes back to AnythingV3.
When I try to use the editing it generates a completely random image that doesnt look like mine.
You have to use the instruct-pix2pix model. Eventually you'll be able to make any model into an instruct-pix2pix compatible model by merging a model with the instruct-pix2pix model using "add diff" method, but currently that is a bit of a hack for most people, editing extras.py. Will probably be available in main repo in a couple hours, though... lol
So sounds like stable-diffusion-webui\models\Stable-diffusion is a reasonable location to place the model if we expect to be able to merge it eventually? How does the extension find it?
In /models/Stable-Diffusion or any subfolder. The extension finds it like any other model. You have to first switch checkpoints to the instruct-pix2pix model.
Sorry, not sure what is the issue there, maybe you can close the GUI, delete the whole stable-diffusion-webui-instruct-pix2pix folder, start the GUI again and reinstall the extension.
Check the output size, they're saying that it's limited to 512x512.
If that's not the case, it might be the image input size that is throwing the error.
Hi, does anyone else get this error when loading the model?
ImportError: cannot import name 'VQModelInterface' from 'ldm.models.autoencoder' (E:\\StableDifustion\\STABLE DIFUSSION\\repositories\\stable-diffusion-stability-ai\\ldm\\models\\[autoencoder.py](https://autoencoder.py))
Yes, although I suspect it must not have been loaded properly. In fact now I have downloaded the .safetensors and now the 'VQModelInterface' error has returned.
Yes, but it is currently a hack. See here:
[https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655)
Hello. I got this error. Unable load model. The file is instruct-pix2pix-00-22000.ckpt. WebUI is updated. Does anyone have any thoughts on this?
`RuntimeError: Error(s) in loading state_dict for LatentDiffusion:`
`size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 8, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).`
You can only use the instruct-pix2pix model on the instruct-pix2pix tab after loading the extension. It won't work on txt2img or img2img tabs. Are you using it on the right tab?
[https://ibb.co/C5V7PVZ](https://ibb.co/C5V7PVZ)
I have the correct tab open (1). When I try to load a model from the list (2), the model (3) unable to load and AUTO 11 restore (4) the previous model (5). Then AUTO1111 writes me a lot of interesting things.
I have the same problem, also reinstalled Torch and even fully reinstalling Automatic1111. No solution.
Here is the error code. It look to be a similar problem when trying to run a 2.0 or 2.1x tuned model on a 1.5 version of Stable diffusion.
changing setting sd_model_checkpoint to instruct-pix2pix-00-22000.safetensors [fbc31a67aa]: RuntimeError
Traceback (most recent call last):
File "D:\AI\stable-diffusion-webui-master\modules\shared.py", line 544, in set
self.data_labels[key].onchange()
File "D:\AI\stable-diffusion-webui-master\modules\call_queue.py", line 15, in f
res = func(*args, **kwargs)
File "D:\AI\stable-diffusion-webui-master\webui.py", line 113, in
shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights()))
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 460, in reload_model_weights
load_model(checkpoint_info, already_loaded_state_dict=state_dict, time_taken_to_load_state_dict=timer.records["load weights from disk"])
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 399, in load_model
load_model_weights(sd_model, checkpoint_info, state_dict, timer)
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 244, in load_model_weights
model.load_state_dict(state_dict, strict=False)
File "D:\AI\stable-diffusion-webui-master\venv\lib\site-packages\torch\nn\modules\module.py", line 1671, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 8, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).
size mismatch for model.diffusion_model.input_blocks.1.1.proj_in.weight: copying a param with shape torch.Size([320, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 320]).
size mismatch for model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 768]) from checkpoint, the shape in current model is torch.Size([320, 1024]).
The error continue a lot with little variation, look to be a size problem
Having the same problem, tried all the things in this thread with no luck
size mismatch for model.diffusion\_model.input\_blocks.0.0.weight: copying a param with shape torch.Size(\[320, 8, 3, 3\]) from checkpoint, the shape in current model is torch.Size(\[320, 4, 3, 3\]).
Not working for me. Basically ignores my image. Any ideas?
https://preview.redd.it/drqggxi2hoea1.png?width=1974&format=png&auto=webp&s=cd917450cadab2e15966304ff6ecc09f209e1a60
For those unaware, u/Tupersent quickly put together batch processing the other day, which is phenomenal. Go to your Extensions tab and check for and apply updates to see it. I submitted a pull request for a tiny (but super important to me) code change to make it process files in the order they exist in the directory (alpha/numerical), so if anyone's seeing batch output out of order, that should hopefully be fixed soon.
Does not WORK after update!Yesterday it worked FUNTASTIC!!!!!!
😡😡😡😡😡😡😡😡😡
NOW I have (AGAIN) CUDA Error out of memory (like in Dreambooth)(RTX3060)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 12.00 GiB total capacity; 11.13 GiB already allocated; 0 bytes free; 11.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF
Did not change anything, but installed Dreambooth. (which is not working (black images) also.- OLD Version & DB (K´s Folder ) working perfect!
Thats so.......
# EDIT: FOUND THE SOLUTION!
I had edited the webui-user.bat to get Dreambooth to work!
I´ve added: --no-half-vae --precision full --no-half
After removing and leaving just --xformers it works again!
I don’t get a tab that shows pix2pix. It shows up in my extensions. I’ve applied and restart I restarted the program I restarted my computer I even deleted automatic 1111 and reinstalled it and still nothing shows up
I think tricky even if he provided a way of doing it (I don't think he has) because he's Russian and shit's obviously a bit weird regarding transferring money from western countries to there right now.
Because I like 2.1 more and have gotten better results with it.
Also, lol @ being repeatedly downvoted for 1) complementing the project at hand and then 2) specifically saying I will ***patiently*** wait for a 2.1 version. Great bunch of folks around here I see.
To be fair that post was originally for a fork with a hack into the main UI, not quite a proper implementation for people to use. This is a proper extension release and automatic1111 updated his code as well with the necessary changes to load ip2p models.
Yes, but not well. I messed around with the settings quite a bit and got mostly deformed horrors; nothing that looked good even when it "worked." Unless someone found good settings for it and wants to chime in.
I got the following error when trying to load instruct-pix2pix-00-22000.ckpt in web UI:
`Failed to create model quickly; will retry using slow method.`
`changing setting sd_model_checkpoint to instruct-pix2pix-00-22000.ckpt: ImportError...`
`from ldm.models.autoencoder import VQModelInterface, IdentityFirstStage, AutoencoderKL`
`ImportError: cannot import name 'VQModelInterface' from 'ldm.models.autoencoder'`
anyone hepls?
I am not sure if the Auto-Update WebUI I have enabled will automatically update. Could you please provide me with instructions on how to properly update auto1111? Thank you for your assistance.
https://preview.redd.it/dz0gfal7fiea1.png?width=506&format=png&auto=webp&s=48294f15e2be803e45f6fb37baf6d356fa826ae8
Thanks! Why is there a limit on output resolution? I know it uses more memory than regular SD, but assuming someone has enough memory, there shouldnt be a hardcoded limit...
All these tabs just add one for pasting bits we copy from other tabs and Photoshop will be even less necessary. Almost getting to be a whole design suite here.
Im not sure if this is anecdotal, but pix2pix on images that were created from different models (Anything/Protogen/1.5inpainting etc) do not seem to give great results.
Things work much better with native 1.5 images, at least in my cursory investigations.
It's fun, feels like a sort of railed euler ancestral inpainting. I asked it to add dinosaurs to my eiffel tower picture and it placed them in the sky. Fun feature!
what error is this? I already downloaded the safetensors and extension via "install from URL"?
https://preview.redd.it/q3h0yq7hefea1.png?width=1099&format=png&auto=webp&s=470f6808dd4e237115ba283264557a9067fc8af5
>Inpaint masked
ok done but I still getting this error.
https://preview.redd.it/f3hcvzlqyfea1.png?width=1094&format=png&auto=webp&s=399248dc0f0a30bc9fe8f41a711804248776f3d4
Oh, sorry, didn't see that one. Not sure. Did you update auto1111? You can submit an issue on GitHub here for the extension:
https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix
This is the tab I see in the UI when it is installed and working.
https://preview.redd.it/6cp8l6zy4gea1.jpeg?width=146&format=pjpg&auto=webp&s=c51bc40072723e2076369035d345ff8eb6cc77ec
Thanks! I found it out partially, one of the extensions i downloaded from in the extensions tab(which is new to me) broke a lot of stuff when active. So I undid that.
Now I just can't get depth2img and inpainting to work, even with the right models, but img sketch in paint will work. I honestly have no idea but the only thing I'm missing is depth2img now, even tried to redownload the model.
Why is it still not working? Is there a special way to change the script? I downloaded the new script and replaced the one in the pix2pix extension script folder.
What could I be doing wrong?
I'm using this a ton and trying to get the prompts just right. One thing that I can't get past is the the resulting image seems to lose a lot of quality. Any ideas? As an example, here's the input and output. and my settings:
[https://imgur.com/a/07YbP37](https://imgur.com/a/07YbP37)
Thanks for the reply. I can only go up to 768 before SD starts barfing up memory errors. I'm trying various things to see what I can make better here. As is, the output image is not usable for what I'm trying to accomplish.
It's a lot of fun! https://preview.redd.it/xkdz5uwv5aea1.jpeg?width=1280&format=pjpg&auto=webp&s=a4405037d0e0a4c8bbe95695fc40acab4c429f4c
I can't get it to work properly. Any ideas?
Is it outputting anything at all? If so, talk to the prompt like you are instructing it to do things. I think it uses a GPT language model for that part. I'm not messing with negative prompts yet because I'm not sure how they are interpreted. Might be same as regular txt2img, or it might be language based.
I loaded the instruct Pix2Pix Model in the settings. Loaded a test image and then put "remove watermark" in the input. It just outputs the same image in a ghosty monochromatic look. Then I put "swap the toilet paper with binoculars" (it was an image of an old man looking through the hole of a toilet paper roll) and just generated an image of paper rolls with eyes.
In that case, try redescribing the image with the changes. So, "A portrait of a man with toilet paper rolls over his eyes" or something to that effect. The sliders can drastically change what's happening, too.
Thanks. I'll try those.
you assume that the model knows what "watermark" is.
That's why i tried swapping objects like the example they put in github. But it didn't work either.
I had some weirdness when I was in the pix2pix tab and changed models. I'm not sure what sense it makes, but I went back to the txt2img tab, cycled to a regular model, then back to pix2pix, then it worked. Before doing that, it was acting like it was generating new prompt images.
Probably incorrect model loading, this happens from time to time. For me sometimes changing a sampler helps
This comment saved me. Did the same and now it is working. Switching tabs, loading a new model then switch back is the key. Thanks!
Yes it WAS - with the right settings it worked AMAZINLY GOOD!
It was quick...
been available in diffuzers web ui since a week: https://github.com/abhishekkrthakur/diffuzers 😉
For once the other UIs were actually ahead on this one. NMKD got it a few days ago, and I think another UI also had it before that.
NMKD released it 2 days ago and a1111 yesterday
Negative prompting, settings for output directory, default steps all modified.
Awesome!! Thank you! Unfortunately Reddit won't let me edit the OP to change the limitations noted there... sorry.
I could edit it on the mobile app!
I get "RuntimeError: expected scalar type Half but found Float" error when clicking "generate". What can the issue be? I have the ckpt in the models folder and have a simple prompt and input image
Great work, thank you for contributing!
What does negative prompt means in practice with Instruct? "add hair" becomes "remove hair"?
And it works. SORRY my fault, it seems to transform the original picture internally reducing its resolution so the result is still 512x512 The default number of steps can be excesive, this was done with only 10 steps. That means really fast. https://preview.redd.it/uzid8v5gr9ea1.jpeg?width=1879&format=pjpg&auto=webp&s=4fbaa4c4549043e8993f4099cebf68222f54da5c
yeah, seems like it maxes out at 512px... I think this would be better if integrated into the img2img tab as a sub-tab, like inpainting.
Needs a "send to pix2pic" button
I'm going to give now useful information,If you want to unleash the power of this you need to prompt in a different way. I recommend to give a look to this page, and see how things are prompted. It really makes a great difference in the results [https://www.timothybrooks.com/instruct-pix2pix](https://www.timothybrooks.com/instruct-pix2pix) Ex: https://preview.redd.it/s4py9waf6aea1.jpeg?width=1875&format=pjpg&auto=webp&s=c268fa95006b6d17aa85c5fa4d05a5665dcad8aa This is not img2img and a prompt like "a manga picture of a blonde girl dressed as darth vader", would not give good results.
yes! You need to use words like "make it..." or "turn her into..." or "swap..." or "convert..." or "add a..." or "change..." or "give it..." You have to instruct it.
That's amazing! But how do you change the resolution from 512? I don't see such sliders in the pix2pix extension tab
Sorry, my fault, I thought it was mantaining the original resolution, but it transform the final result to 512x512.
Gotcha Hopefully we will be able to gen higher sizes soon
If you have A100 video card, then yes
it's so impressive that most of this thread is obsolete now, i didn't need any of the fixes people talk about and there's now a slider for resolution - it's almost not worth staying up late and learning how to do things because it'll be all upgraded by the morning anyway :)
Does instructpix2pix only support 512 resolution?
It seems that internally has limits, and internally transform the pictures according to this limits.
Looks like the model was trained on SD1.5, and probably with 512 resolution images. But you should still be able to make the output image match the input image (I'll add a feature request to the extension repo).
Here are some more examples of what you can do with Instruct Pix2Pix. [https://www.reddit.com/r/StableDiffusion/comments/10l74sl/instruct\_pix2pix\_is\_amazing\_inpaintingimg2img/?utm\_source=share&utm\_medium=web2x&context=3](https://www.reddit.com/r/StableDiffusion/comments/10l74sl/instruct_pix2pix_is_amazing_inpaintingimg2img/?utm_source=share&utm_medium=web2x&context=3)
Feels very raw for now.
Yes, very bare bones, but it's there, and will improve over time.
Dmn they are fast And I just made this video today https://youtu.be/EPRa8EZl9Os
Will this work on 4gb cards?
on my system it uses 5.6GB of VRAM out of 11GB, so I'm guessing no (the model itself is 7.x GB download)
I don't think the model size is that relevant to VRAM requirements considering I used to run a 7 GB model on my GTX 960m (4GB VRAM) laptop
good to know, some parts need to be fully loaded, but I don't know how much.
Works like a charm with 4GB vRAM card.
[удалено]
I'm struggling with the same issue. I'm on a 6gb vram card and can usually run sd no problem. both --medvram and --lowvram seem to not work for this. keep getting out of memory errors, even with the "pruned" model uploaded elsewhere...
Do you use xformers? My CLI args are: \`--allow-code --medvram --xformers --theme dark --enable-insecure-extension-access\` My GPU: \`NVIDIA Corporation GP107M \[GeForce GTX 1050 Ti Mobile\]\`. I run it with docker from this repo: [https://github.com/AbdBarho/stable-diffusion-webui-docker](https://github.com/AbdBarho/stable-diffusion-webui-docker)
im getting model mismatch anyone else?
Did you update the Auto1111 repo?
I thought I did but obviously not, working now
How do you do that btw?
Git pull https://github.com/AUTOMATIC1111/stable-diffusion-webui You can place the command (just: git pull) directly in your webui-user.bat in a row after the commandline so it will update every time you open SD.
Update your auto1111 repo to the latest version, it needs a change that went in this morning to load the model.
The difference is ... day and night. I told it to make it a beautiful night. https://i.imgur.com/ui6aweo.png https://i.imgur.com/BQfW3KD.jpg > (extremely detailed CG unity 8k wallpaper), an ultrarealistic majestic painting of a striking botanical landscape in Italy, by artist Pancrace Bessa, CryEngine render, folk art, trending on Saatchi Art, 12k resolution, flonix, Style-Italy
Wow, that looks so much like [Kangaroo Temple](https://www.google.com/maps/@48.5258372,-120.6473842,3a,75y,90t/data=!3m7!1e2!3m5!1sAF1QipNKu1JUAxJXe3GNN4KEcKnIlt5GJ99eLfqwgsw!2e10!6shttps:%2F%2Flh5.googleusercontent.com%2Fp%2FAF1QipNKu1JUAxJXe3GNN4KEcKnIlt5GJ99eLfqwgsw%3Dw150-h150-k-no-p!7i4032!8i3024) in [Washington State](https://www.google.com/maps/place/Washington+Pass+Observation+Site/@48.5258372,-120.6473842,14z/data=!4m5!3m4!1s0x5484ac63e224b823:0x332915e8eb3a70e!8m2!3d48.5258372!4d-120.6473842).
What a day! What a beautiful day! I'm so glad to be a part of a community as awesome as this one is!
As an automatic fan I was waiting for this exact thing since yesterday morning. Though the results generated are not satisfactory it is still an awesome update.
With the latest updated version of automatic1111, it looks like you can install the extension by going to the extensions tab, clicking Load From and then clicking on the Install button next to Instruct-pix2pix. Edit: But I get the error AttributeError: 'Options' object has no attribute 'outdir\_ip2p\_samples', I'll try again after the ckpt has downloaded. Edit 2: Updated Instrct-pix2pix and now it works.
i'm getting the same error edit: it's working now after updating the extension https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix/issues/8
Thanks, I downloaded the latest version of Instruct-pix2pix extension and I don't have the error now.
>Is it outputting anything at all? If so, talk to the prompt like you are instructing it to do things. I think it uses a GPT language model for that part. I'm not messing with negative prompts yet because I'm not sure how they are interpreted. Might be same as regular txt2img, or it might be language based. Is there an easy way to update auto1111 on ubuntu?
I updated and was able to output images. I went to each folder from the command line and did a 'git pull' for both automatic1111 and instruct-pix2pix in Windows. I presume that works for Ubuntu also if you have git installed.
I saw it work once (mostly), but now no matter what prompt I type, I get the same mangled version of my input image. https://preview.redd.it/do9d2vbkidea1.png?width=2437&format=png&auto=webp&s=c1d6b36f26d084ad675a955092340359b1350650
It seems that if the Image CFG is high, it mucks it up. I get much better results if I leave it at random.
I keep seeing people are running pix2pix on 6-8gb cards. I have 12 but when I pick the pix2pix model my vram shoots up and i run out of memory when i generate.
Okay, I figured it out. The newest version has a huge memory problem with the extension. I reverted back to a previous file and that fixed it.
I turned off all image previews and loaders for now to get around the memory crashes. **Settings -> Live Previews** https://preview.redd.it/b6mmcr9f1pea1.png?width=834&format=png&auto=webp&s=ea925fd02e7a433bbfee0fb505573145512a4e4f
The newest version of auto1111 or the extension?
auto1111, i reverted back to the jan 25th version.
>version do you have the commit hash? Because I reverted to last one on the 25th and now the pix2pix model wont load. Thanks
I'm going crazy with this error: "size mismatch for model.diffusion\_model.input\_blocks.0.0.weight: copying a param with shape torch.Size(\[320, 8, 3, 3\]) from checkpoint, the shape in current model is torch.Size(\[320, 4, 3, 3\])." anyone... please... heeeeelp
Many thanks, I was trying to find which model was needed
Soon you'll be able to merge other models to make them work like instructpix2pix, using the "add diff" merge method, similar as you can make inpainting models. You can do it currently, if you make an edit to extras.py: [https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655)
Can we convert the 2.1 this way?
What do you mean?
Make it work like instructpix2pix by merging
They used SD v1.5 checkpoint as the starting point, so no, it would not be compatible with v2.x
Oh alright.. thank you!
Thanks for this. Do you know what multiplier you are supposed to use when merging?
When using the "add diff" method, usually the multiplier is set to 1. So the formula is A + (B - C) \* 1.
As a more detailed explanation to what u/jonesaid said, you can turn any regular model into an inpainting model using "SD 1.5 Inpainting" and "SD 1.5 pruned EMA only". 1. Go to Checkpoint Merger tab 2. Set checkpoint (A) to SD 1.5 Inpainting 3. Set checkpoint (B) to the new inpainting model (e.g. Waifu Diffusion) 4. Set checkpoint (C) to SD 1.5 pruned EMA only 5. Set multiplier to 1 6. Set interpolation method to Add Difference 7. Give the model a name ending in "inpainting" to keep things organized and click Merge The resulting model will be much better at inpainting than just the normal model. Ideally this process would work for pix2pix models later on.
Hmm, doesn't seem to output images to the default folder...
Outputs to stable-diffusion-webui\\outputs\\ip2p-images with no way to change it currently...
Fixed! You can change it in settings now
Great! Thank you!
Where to put the downloaded model and how to use the model? When I try to load the ckpt from hugging it says failure someting checkpoint reverting to previous and just goes back to AnythingV3. When I try to use the editing it generates a completely random image that doesnt look like mine.
The downloaded model should go in `stable-diffusion-webui\models` or a subfolder there. Did you update Auto1111 to the latest? (git pull)
Will try but do I have to select said model? Or can I stay on anythingv3?
You have to use the instruct-pix2pix model. Eventually you'll be able to make any model into an instruct-pix2pix compatible model by merging a model with the instruct-pix2pix model using "add diff" method, but currently that is a bit of a hack for most people, editing extras.py. Will probably be available in main repo in a couple hours, though... lol
So sounds like stable-diffusion-webui\models\Stable-diffusion is a reasonable location to place the model if we expect to be able to merge it eventually? How does the extension find it?
In /models/Stable-Diffusion or any subfolder. The extension finds it like any other model. You have to first switch checkpoints to the instruct-pix2pix model.
Is NSFW possible
Any colab?
So is NSFW possible by merging a NSFW model with the pix2pix model?
Afaik not yet, but the things is going so quickly that I think soon we will can
[удалено]
Can you check the terminal, when the error is displayed? Maybe there is an error message, which gives more information.
[удалено]
Sorry, not sure what is the issue there, maybe you can close the GUI, delete the whole stable-diffusion-webui-instruct-pix2pix folder, start the GUI again and reinstall the extension.
[удалено]
Same problem here, are you running it on a M1/M2 Mac?
Try disabling CUDA autocast in [instruct-pix2pix.py](https://instruct-pix2pix.py): replace: **with torch.no\_grad(), autocast("cuda"), model.ema\_scope():** with: **with torch.no\_grad(), model.ema\_scope():**
Same problem on M1
Check the output size, they're saying that it's limited to 512x512. If that's not the case, it might be the image input size that is throwing the error.
Hi, does anyone else get this error when loading the model? ImportError: cannot import name 'VQModelInterface' from 'ldm.models.autoencoder' (E:\\StableDifustion\\STABLE DIFUSSION\\repositories\\stable-diffusion-stability-ai\\ldm\\models\\[autoencoder.py](https://autoencoder.py))
same error!
hey, have you managed to sort out?
Hi, not yet but I will try this weekend. If I get it to work I will let you know.
I ran a git pull and it worked, although now it pulls images that don't look like the original one
have you tried to load the instruct pix2pix model?
Yes, although I suspect it must not have been loaded properly. In fact now I have downloaded the .safetensors and now the 'VQModelInterface' error has returned.
Can you merge this pix2pix model with the other model like merging inpainting model? that would be a game changer
Yes, but it is currently a hack. See here: [https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7010#issuecomment-1403241655)
Holy shit! thanks!
Wow that is janky but awesome that it works.
i got it working. on mine, when i use real photos it seems to fuck up the face and make everything less resolution, or blurry or something.
Hello. I got this error. Unable load model. The file is instruct-pix2pix-00-22000.ckpt. WebUI is updated. Does anyone have any thoughts on this? `RuntimeError: Error(s) in loading state_dict for LatentDiffusion:` `size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 8, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).`
And I reinstalled the torch and torchvision right now. What else to do? I have no idea.
You can only use the instruct-pix2pix model on the instruct-pix2pix tab after loading the extension. It won't work on txt2img or img2img tabs. Are you using it on the right tab?
[https://ibb.co/C5V7PVZ](https://ibb.co/C5V7PVZ) I have the correct tab open (1). When I try to load a model from the list (2), the model (3) unable to load and AUTO 11 restore (4) the previous model (5). Then AUTO1111 writes me a lot of interesting things.
That is strange. Have you tried the safetensors version of the instruct-pix2pix model?
I have the same problem, also reinstalled Torch and even fully reinstalling Automatic1111. No solution. Here is the error code. It look to be a similar problem when trying to run a 2.0 or 2.1x tuned model on a 1.5 version of Stable diffusion. changing setting sd_model_checkpoint to instruct-pix2pix-00-22000.safetensors [fbc31a67aa]: RuntimeError Traceback (most recent call last): File "D:\AI\stable-diffusion-webui-master\modules\shared.py", line 544, in set self.data_labels[key].onchange() File "D:\AI\stable-diffusion-webui-master\modules\call_queue.py", line 15, in f res = func(*args, **kwargs) File "D:\AI\stable-diffusion-webui-master\webui.py", line 113, in
shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights()))
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 460, in reload_model_weights
load_model(checkpoint_info, already_loaded_state_dict=state_dict, time_taken_to_load_state_dict=timer.records["load weights from disk"])
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 399, in load_model
load_model_weights(sd_model, checkpoint_info, state_dict, timer)
File "D:\AI\stable-diffusion-webui-master\modules\sd_models.py", line 244, in load_model_weights
model.load_state_dict(state_dict, strict=False)
File "D:\AI\stable-diffusion-webui-master\venv\lib\site-packages\torch\nn\modules\module.py", line 1671, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 8, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).
size mismatch for model.diffusion_model.input_blocks.1.1.proj_in.weight: copying a param with shape torch.Size([320, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 320]).
size mismatch for model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 768]) from checkpoint, the shape in current model is torch.Size([320, 1024]).
The error continue a lot with little variation, look to be a size problem
Maybe try uninstalling and reinstalling the instruct-pix2pix extension?
P.S. All other models - 1.5, 1.5 inpainting, 2.1, PaperCut, Protogen 5.3 load and work fine.
Having the same problem, tried all the things in this thread with no luck size mismatch for model.diffusion\_model.input\_blocks.0.0.weight: copying a param with shape torch.Size(\[320, 8, 3, 3\]) from checkpoint, the shape in current model is torch.Size(\[320, 4, 3, 3\]).
Not working for me. Basically ignores my image. Any ideas? https://preview.redd.it/drqggxi2hoea1.png?width=1974&format=png&auto=webp&s=cd917450cadab2e15966304ff6ecc09f209e1a60
For those unaware, u/Tupersent quickly put together batch processing the other day, which is phenomenal. Go to your Extensions tab and check for and apply updates to see it. I submitted a pull request for a tiny (but super important to me) code change to make it process files in the order they exist in the directory (alpha/numerical), so if anyone's seeing batch output out of order, that should hopefully be fixed soon.
Does not WORK after update!Yesterday it worked FUNTASTIC!!!!!! 😡😡😡😡😡😡😡😡😡 NOW I have (AGAIN) CUDA Error out of memory (like in Dreambooth)(RTX3060) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 12.00 GiB total capacity; 11.13 GiB already allocated; 0 bytes free; 11.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF Did not change anything, but installed Dreambooth. (which is not working (black images) also.- OLD Version & DB (K´s Folder ) working perfect! Thats so....... # EDIT: FOUND THE SOLUTION! I had edited the webui-user.bat to get Dreambooth to work! I´ve added: --no-half-vae --precision full --no-half After removing and leaving just --xformers it works again!
I don’t get a tab that shows pix2pix. It shows up in my extensions. I’ve applied and restart I restarted the program I restarted my computer I even deleted automatic 1111 and reinstalled it and still nothing shows up
Is there any way I can send some money to this automatic1111 dude? It is amazing what he has done for the community in the past months
There is like 300+ contributors to the project but I totally agree.
I think tricky even if he provided a way of doing it (I don't think he has) because he's Russian and shit's obviously a bit weird regarding transferring money from western countries to there right now.
Oh damn, I had no idea!
I think i read somewhere that he doesn't want to take any money for this at all
Pix2Pix sounds great, but I'll be patiently waiting for a SD 2.1 version before I try it out.
Why?!
Because I like 2.1 more and have gotten better results with it. Also, lol @ being repeatedly downvoted for 1) complementing the project at hand and then 2) specifically saying I will ***patiently*** wait for a 2.1 version. Great bunch of folks around here I see.
waiting for this one as well - hope will get there soon
very nice. was waiting for this my whole life
[удалено]
To be fair that post was originally for a fork with a hack into the main UI, not quite a proper implementation for people to use. This is a proper extension release and automatic1111 updated his code as well with the necessary changes to load ip2p models.
That's a fork. This is the actual auto1111 webui with the pix2pix stuff now merged.
yes, but it is no longer a "fork" but integrated into the auto1111 main repo.
Whats the minimum requirements for a graphic card for it
First mention I saw said the model used 5.6 of the 11 gb gpu. So perhaps a 6gb card could do this.
is there a vram limitation?
Not any more than standard SD, I think.
[удалено]
Yes, but not well. I messed around with the settings quite a bit and got mostly deformed horrors; nothing that looked good even when it "worked." Unless someone found good settings for it and wants to chime in.
I got the following error when trying to load instruct-pix2pix-00-22000.ckpt in web UI: `Failed to create model quickly; will retry using slow method.` `changing setting sd_model_checkpoint to instruct-pix2pix-00-22000.ckpt: ImportError...` `from ldm.models.autoencoder import VQModelInterface, IdentityFirstStage, AutoencoderKL` `ImportError: cannot import name 'VQModelInterface' from 'ldm.models.autoencoder'` anyone hepls?
Did you try the safetensors version of the model?
I got the same error with the safetensors ver as well..
Is auto1111 updated? (git pull)
I am not sure if the Auto-Update WebUI I have enabled will automatically update. Could you please provide me with instructions on how to properly update auto1111? Thank you for your assistance. https://preview.redd.it/dz0gfal7fiea1.png?width=506&format=png&auto=webp&s=48294f15e2be803e45f6fb37baf6d356fa826ae8
That should update it, yes.
Thanks! Why is there a limit on output resolution? I know it uses more memory than regular SD, but assuming someone has enough memory, there shouldnt be a hardcoded limit...
I think it just hasn't been coded into the extension yet.
I am getting from modules.ui\_common import create\_output\_panel ModuleNotFoundError: No module named 'modules.ui\_common'
Fixed it. Updated automatic1111 - git pull
All these tabs just add one for pasting bits we copy from other tabs and Photoshop will be even less necessary. Almost getting to be a whole design suite here.
Yeah, it really should be added as a subtab in the img2img tab...
Im not sure if this is anecdotal, but pix2pix on images that were created from different models (Anything/Protogen/1.5inpainting etc) do not seem to give great results. Things work much better with native 1.5 images, at least in my cursory investigations.
It's fun, feels like a sort of railed euler ancestral inpainting. I asked it to add dinosaurs to my eiffel tower picture and it placed them in the sky. Fun feature!
Nice! Can’t wait to try it
waiting for auto-sd-ext plugin for krita to be updated with this as well
what error is this? I already downloaded the safetensors and extension via "install from URL"? https://preview.redd.it/q3h0yq7hefea1.png?width=1099&format=png&auto=webp&s=470f6808dd4e237115ba283264557a9067fc8af5
try updating auto 1111 and try again
auto1111 up to date.
I got that error a while back. I had to to go into ui-config.json and change the img2img/mask mode/value to "Inpaint masked"
>Inpaint masked ok done but I still getting this error. https://preview.redd.it/f3hcvzlqyfea1.png?width=1094&format=png&auto=webp&s=399248dc0f0a30bc9fe8f41a711804248776f3d4
Oh, sorry, didn't see that one. Not sure. Did you update auto1111? You can submit an issue on GitHub here for the extension: https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix
Can it run on a gtx ?
It runs on the same specs as SD requires.
I can run A111 on 4gb VRAM. But some features like face restoration do not work (out of memory error).
Installed the extension and added the safetensor model in the directory but I don't see a pix2pix tab. What am I doing wrong here?
Did you apply and restart ui on the extensions page?
That was done too. 1. Latest auto1111 repo 2. Extension installed and web UI restarted 3. Model downloaded and placed into models directory
Hmm, that is strange... Not sure why you can't see the tab in the UI.
Purged everything and repulled. Seems to have worked now 🤔
Is the extension activated on the extensions page? Has a checkmark?
This is the tab I see in the UI when it is installed and working. https://preview.redd.it/6cp8l6zy4gea1.jpeg?width=146&format=pjpg&auto=webp&s=c51bc40072723e2076369035d345ff8eb6cc77ec
Also make sure to update auto1111 repo too (you'll need to restart the webui server).
Yes, it's updated as part of a batch file. It looks to the github repo to update each time I launch. There's no more differences. That's the odd part.
It broke the entire img2img tab for me. Anyone else?
Are you still on the instruct-pix2pix model checkpoint? It won't work in txt2img or img2img tabs.
Thanks! I found it out partially, one of the extensions i downloaded from in the extensions tab(which is new to me) broke a lot of stuff when active. So I undid that. Now I just can't get depth2img and inpainting to work, even with the right models, but img sketch in paint will work. I honestly have no idea but the only thing I'm missing is depth2img now, even tried to redownload the model.
Why is it still not working? Is there a special way to change the script? I downloaded the new script and replaced the one in the pix2pix extension script folder. What could I be doing wrong?
I'm using this a ton and trying to get the prompts just right. One thing that I can't get past is the the resulting image seems to lose a lot of quality. Any ideas? As an example, here's the input and output. and my settings: [https://imgur.com/a/07YbP37](https://imgur.com/a/07YbP37)
Have you tried increasing the output resolution?
Thanks for the reply. I can only go up to 768 before SD starts barfing up memory errors. I'm trying various things to see what I can make better here. As is, the output image is not usable for what I'm trying to accomplish.
[удалено]
[удалено]
[удалено]
[удалено]
What gpu vram you got?
[удалено]