There is a lot of overhead in starting up pools and transferring work over to them. It is common that parallel is slower unless you give it "enough" work to do.
Also, in newer versions, some parts of MATLAB transparently use multiple cores for sufficiently large computations, even without the parallel toolbox. If those cores are not available because they are reserved for the pool, then the execution could slow down.
Just opening a pool does not trigger parallel computation: routines do not, usually, take advantage of an available pool, perhaps figuring that you must have something else in mind for the pool. You explicitly code parallelization with parfor or spmd or similar routines -- except, that is, for the parts that automatically use multiple cores even if you do not have a pool.
