I'm not familiar with vtex and image editing with layers.
I can make normal textures, but when I tried to modify original signages, it showed up as two layer ingame, in the background the original graphic was visible. In my graphic editor program I didnt see any lyer in the tga, so I was forced to save it as jpg to get rid of any hidden thing, Then the vtf size became half, but then the signages appeared as normal overlays without the white selflight effect. So...
How can I create overlay textures, and what settings I must check in VTFedit when importing/saving?

You just make a .tga file with an Alpha channel that is white on that parts you want to glow and black on the parts that shouldn't (like the text, stick man, border). Run this through vtex normally and the .VMT should be:

"LightmappedGeneric"
{
"$basetexture" "texturelocationhere"
"$decal" 1
"$surfaceprop" "glass"
"$selfillum" 1
"%keywords" "portal"
}