Big ships will go to the west coast. You won't be able to sink a single one. They will be out of range. The big ports of India do more then 20000 ships a year altogether. 15-20 is actually less than 0.1%.
Whether it goes to east coast or west coast, doesn't matter. Because it has to pass through there. Over 55% india's maritime trade passes through Malacca straight. We would definitely sink many of them. And we don't have to necessarily get inside Malacca. We could definitely ambush in and around. I said 15-20 large ships as the opening salvo. (Not the end state.) Each carrying $300-400 millions worth good on average. That would be enough to deter many shipping companies.
An AIP sub travels at 5 knots and has around 20 days endurance. That is around 4500 km. If you are doing the full distance using AIP, you will not reach Malacca. You will not be going in a straight line and you will need some amount of charge to prowl around searching for victims. You could run the starting 500 km and the last 500km on the return journey on diesel engine, but that would be dangerous.
You don't expect BN to be equipped with AIP subs only. There is an ongoing frigate program. Given we are talking hypotheticals, those ships can be utilized togather for a degree of escort as they won't Go on any offensive. Also, I don't understand why did you say BN sub can't reach Malacca? It's around 1900-2000km if we go a bit zigzag. Less distance in straight travel.
As for submarines needing to return to base after running out of AIP fuel, well, this is not so straight forward. lithium ion battery equipped AIP boats are now becoming available. thus new boats can recharge at periscope depth after 15 days of being submerged, dive again for another 15 and resurface again. It can repeat such process up to 3 times. Basically, augmented with lithium ion battery, it can conserve its AIP fuels and use it economically in stages. (Same process can also be replicated with Lead-acid batteries equipped AIP boats. Although interval periods bwtween resurfacing would be smaller.) In such cases, BD submarines won't return to bases, Instead they will go further down into open ocean, recharge and come back again. For exmaple, a type 214 has 80 days of total endurance.
Yes, there will be risk in such tactics. But military ops always comes with high danger. Risk acceptance is part of the job.
Even if the sub reached the Malacca shipping lane, when the submarine comes to periscope depth, what it will see is a lot of ships. There is no knowing which one to hit. None of them will have the Indian flag. Any information the captain has will be 10 days old.
No, modern submarine have sensitive enough sensor system that can differentiate a ship down to its individual signeture (not just the model or variant) from hundreds of miles away. These are continuesly prerecorded in peacetimes to utilized in conflict. No need to go to at periscope depth to fire torpedos like ww2.